telepresenceio / telepresence Goto Github PK

Setting TMPDIR=/tmp seems to fix it; default appears not to be shared well by Docker.

We should switch to Netlify (and enable HTTPS) instead of Github Pages since if we can figure out how to make it only build on tagged builds. Or perhaps push to netlify from Travis in the release process, instead of relying on Netlify's github hooks

--proxy only works for domain names at the moment.

exarkun@baryon:~/Work/LeastAuthority/leastauthority.com$ ../telepresence --deployment s4-infrastructure --run-shell
Starting proxy...
exarkun@baryon:~/Work/LeastAuthority/leastauthority.com$ logout
Shutting proxy down...
exarkun@baryon:~/Work/LeastAuthority/leastauthority.com$ 

It does appear in my gnome-terminal / tmux / bash sessions.

Some limitations that aren't currently documented and should be:

• Custom DNS resolver libraries won't work (UDP isn't proxied at the moment).
• Statically linked binaries won't work. This is the default for Go - workaround is gccgo + specific env variable, see docs for details.
• suid binaries won't work.

CC @askcarter

kubectl exec will connect to a random container...

What were you trying to do?

Start a telepresence shell. I have the SSH option "Compression yes" set in ~/.ssh/config.

What did you expect to happen?

The shell would start.

What happened instead?

There is a long pause after "Starting proxy...", then the traceback below is printed. If I removed the Compression setting from my SSH config, the command is successful.

Automatically included information

Command line: ['/Users/ssorensen/bin/telepresence', '--new-deployment', 'quickstart', '--run-shell']
Version: 0.26
Python version: 3.6.0 (default, Mar 27 2017, 20:59:04) [GCC 4.2.1 Compatible Clang 3.7.1 (tags/RELEASE_371/final)]
kubectl version: Client Version: v1.5.3-dirty
OS: Darwin 200956 16.5.0 Darwin Kernel Version 16.5.0: Fri Mar 3 16:52:33 PST 2017; root:xnu-3789.51.2~3/RELEASE_X86_64 x86_64 i386 MacBookPro12,1 Darwin
Traceback:

Traceback (most recent call last):
  File "/Users/ssorensen/bin/telepresence", line 678, in call_f
    return f(*args, **kwargs)
  File "/Users/ssorensen/bin/telepresence", line 761, in go
    subprocesses, env, socks_port = start_proxy(runner, args)
  File "/Users/ssorensen/bin/telepresence", line 507, in start_proxy
    args.expose,
  File "/Users/ssorensen/bin/telepresence", line 433, in connect
    wait_for_ssh(runner, ssh_port)
  File "/Users/ssorensen/bin/telepresence", line 379, in wait_for_ssh
    raise RuntimeError("SSH isn't starting.")
RuntimeError: SSH isn't starting.

Logs:

le=/dev/null', 'root@localhost', '/bin/true'],)
Running: (['ssh', '-q', '-p', '60325', '-oStrictHostKeyChecking=no', '-oUserKnownHostsFile=/dev/null', 'root@localhost', '/bin/true'],)
Running: (['ssh', '-q', '-p', '60325', '-oStrictHostKeyChecking=no', '-oUserKnownHostsFile=/dev/null', 'root@localhost', '/bin/true'],)
Running: (['ssh', '-q', '-p', '60325', '-oStrictHostKeyChecking=no', '-oUserKnownHostsFile=/dev/null', 'root@localhost', '/bin/true'],)
Running: (['ssh', '-q', '-p', '60325', '-oStrictHostKeyChecking=no', '-oUserKnownHostsFile=/dev/null', 'root@localhost', '/bin/true'],)
Running: (['ssh', '-q', '-p', '60325', '-oStrictHostKeyChecking=no', '-oUserKnownHostsFile=/dev/null', 'root@localhost', '/bin/true'],)
Running: (['ssh', '-q', '-p', '60325', '-oStrictHostKeyChecking=no', '-oUserKnownHostsFile=/dev/null', 'root@localhost', '/bin/true'],)
Running: (['ssh', '-q', '-p', '60325', '-oStrictHostKeyChecking=no', '-oUserKnownHostsFile=/dev/null', 'root@localhost', '/bin/true'],)

Keeping them around is confusing.

You need to listen on 0.0.0.0 to get exposed to Kubernetes.

Removing dependency on Docker will make this easier to fix.

Telepresence shouldn't need much memory or CPU inside k8s, so limit resources requested so scheduler doesn't evict stuff on its behalf.

At this point Telepresence doesn't actually require Docker, and on Linux it means we need sudo. We should be able to remove the dependency by moving code in entrypoint.py into the top-level CLI (cli/telepresence).

• Document new requirements.
• Remove sshpass requirement.
• Assign random ports for local SSH.
• Assign random ports for proxied SOCKS.
• Test on Mac.

This results in failure to start up.

When I use --new-shell in a tmux session, if I type a command that reaches the edge of the terminal, the illusion of a teletype is shattered by incorrect linewrapping behavior.

out.ogv.zip

• Kubernetes cluster in the cloud.
• Build docker images for git hash, not latest version, except when doing actual release.
• Tests don't use any shared resources (need to add more randomness, etc..)
• CI runs tests using custom generated images.

If docs use --verbose flag, it'll make initial expectation of speed clearer, without cluttering up default display.

Currently telepresence uses the currently set k8s context. It should be possible to specify it on the command line:

$ telepresence --k8s-context=minikube ...

Sometimes SSHing into remote pod fails because kubectl port-forward takes a while. We should loop and retry, rather than relying on hardcoded sleep.

--new-deployment searches for Deployment and Pod by name... but that may be left over from a previous run, e.g. if it's still shutting down.

It would be better to set some unique per-run metadata and use that to find them, rather than name.

Nothing notices that it exits!

Currently each release involves updating download URLs. We should have a URL that downloads latest version.

If the connection is lost, telepresence doesn't reconnect, the user needs to restart it completely.

What were you trying to do?

Following the instructions on https://datawire.github.io/telepresence/ I was trying to run a local container with telepresence and have it hooked in to an existing deployment which I had added the telepresence container to on my Kubernetes cluster.

I got to the ./telepresence --deployment ... command in "2. Run the local Telepresence client on your machine" and then encountered an exception.

What did you expect to happen?

To have my container started up locally and plumbed into the networking of the pod for the selected deployment.

What happened instead?

Traceback for docker run error, no running container.

Automatically included information

Command line: ['./telepresence', '--deployment', 's4-infrastructure', '--docker-run', 'leastauthority/grid-router']
Version: 0.7
Python version: 2.7.12+ (default, Sep 17 2016, 12:08:02) [GCC 6.2.0 20160914]
OS: Linux baryon 4.8.0-37-generic #39-Ubuntu SMP Thu Jan 26 02:27:07 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
Traceback:

Traceback (most recent call last):
  File "./telepresence", line 197, in call_f
    return f(*args, **kwargs)
  File "./telepresence", line 227, in go
    tempdir, container_id = start_proxy(args)
  File "./telepresence", line 139, in start_proxy
    container_id = unicode(check_output(docker_cmd).strip(), "utf-8")
  File "/usr/lib/python2.7/subprocess.py", line 574, in check_output
    raise CalledProcessError(retcode, cmd, output=output)
CalledProcessError: Command '['docker', 'run', '--detach', '--rm', '-v', '/home/exarkun:/opt:ro', '-v', '/home/exarkun:/home/exarkun:ro', '-v', '/tmp/tmpjWD__K:/output', 'datawire/telepresence-local:0.7', '1000', 's4-infrastructure', '', '']' returned non-zero exit status 1

Building images is taking decent chunk of CI time (160 seconds). Could probably be much faster if we used Alpine, say.

In the guestbook example directory, I ran

$ telepresence --deployment telepresence-deployment --expose 8080 --run-shell
Starting proxy...

... and it is sitting there.

Log file says:

Running: (['docker', 'run', '--rm', '--name', 'telepresence-1491325324-79-46205', '-t', '-v', '/Users/ark3:/opt:ro', '-v', '/Users/ark3:/Users/ark3:ro', '-p', ':9050', '-v', '/tmp/tmpQhUaSJ:/output', 'datawire/telepresence-local:0.23', '501', 'telepresence-deployment', '8080', '10.12.108.229', 'null'],)
Unable to find image 'datawire/telepresence-local:0.23' locally
0.23: Pulling from datawire/telepresence-local
0a8490d0dfd3: Pulling fs layer
51f9c8ec365f: Pulling fs layer
00f17031bf5f: Pulling fs layer
00f17031bf5f: Verifying Checksum
00f17031bf5f: Download complete
0a8490d0dfd3: Verifying Checksum
0a8490d0dfd3: Download complete
0a8490d0dfd3: Pull complete
51f9c8ec365f: Verifying Checksum
51f9c8ec365f: Download complete
51f9c8ec365f: Pull complete
00f17031bf5f: Pull complete
Digest: sha256:96c6330bacca0c60c521ac2a4f8ed829574db665dac9c05824a444960069c9dd
Status: Downloaded newer image for datawire/telepresence-local:0.23
Unable to connect to the server: dial tcp 192.168.64.5:8443: i/o timeout
Traceback (most recent call last):
  File "/usr/local/bin/entrypoint.py", line 348, in <module>
    loads(argv[5])
  File "/usr/local/bin/entrypoint.py", line 320, in main
    remote_info = get_remote_info(deployment_name, namespace)
  File "/usr/local/bin/entrypoint.py", line 153, in get_remote_info
    "--export",
  File "/usr/lib/python3.5/subprocess.py", line 626, in check_output
    **kwargs).stdout
  File "/usr/lib/python3.5/subprocess.py", line 708, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['kubectl', 'get', 'deployment', '-o', 'json', 'telepresence-deployment', '--export']' returned non-zero exit status 1

Telepresence would benefit from just enough volume support to allow reading things like secrets, configmaps, Downward API, and the like. Basically, read-only access.

Transparent options:

• https://github.com/proot-me/PRoot - Linux only, doesn't require root, process-specific.
• FUSE, which appears to be available on Linux and Mac, plus https://github.com/rpodgorny/unionfs-fuse or http://bindfs.org/. Might require root? Also impacts all processes.
• AppImage/AppImageKit#267 has some more relevant discussion.
• NixOS/nix#361 has some mac-specific discussion.
• OS X supports directory hard links. So some combination of hard links and symlinks to simulate base system? dunno.
• http://unix.stackexchange.com/a/81027 (unshare) + bind mounts (Linux only). This is probably the best option, bind mounts and namespaces are built-in Linux functionality.

Manual options:

• Special env var $MOUNTS_ROOT is set to / in production, a temporary directory in Telepresence. Users must manually use it as basis for paths.

Explain all the cool/horrible tricks Telepresence uses to work.

This is because of torsocks, just need to disable that option.

Originally reported by @exarkun .

A Docker image may hardcode that it needs to connect to a Service listening on port 6379. Telepresence might proxy that via port 2001, though.

A solution which can also be used to solve #6: each destination gets its own loopback IP, e.g. 127.0.0.2. You can then listen on same port as destination without worrying about conflicts or the like.

@richarddli reported this.

This can happen when --deployment is used and user forgets to upgrade.

If the Docker server is not on the local machine there all kinds of fun ways telepresence can break. Complain if we're using non-local Docker server, i.e. DOCKER_HOST env variable is set.

1. Run tests (minikube start; make minikube-test)
2. Run partial tests (TELEPRESENCE_TESTS="-k tocluster tests/test_run.py" make minikube-test)
3. Run command line with images built above (TELEPRESENCE_VERSION=$(make version) cli/telepresence ...)

1. The better way to query a Pod rather than using grep is kubectl get pod --selector=run=helloworld ... recommend changing this kubectl get pod | grep helloworld which might be confusing potentially if using a shared cluster.
2. switching to / # wget -qO- http://quickstart.default.svc.cluster.local:8080/file.txt since its bundled on busybox and you don't need to have them instal lcurl
3. Table of contents (probably w/https://github.com/dafi/tocmd-generator until we have better documentation system)

Currently telepresence runs local code in a container. It would be even nicer if it didn't require that and code could be run normally on the developer machine directly.

Interested in this feature? Add a "thumbs up" or comment below.

Implementation options:

• torsocks - LD_PRELOAD a library that redirects network traffic over a SOCKS proxy; the latter would be done using SSH as first pass.
• http://sshuttle.readthedocs.io/en/stable/requirements.html - this would require root, so I'm not sure I want to pursue it, but worth mentioning.

I tried using the uncontainerized process support in 0.14 and the first line of output when my app starts up now is:

[Mar 21 11:17:26] ERROR torsocks[19467]: Unable to resolve. Status reply: 4 (in socks5_recv_resolve_reply() at socks5.c:666)

I don't know if this is a critical error preventing my app from networking properly or if it's just a warning that something non-critical that failed that I can safely ignore.

One way of specifying an alternate context for kubectl is to set the environment variable KUBECONFIG to point to a non-default configuration file. Once done, this allows the user to use kubectl as always to talk to the desired cluster.

Telepresence does not respect this environment variable, leading to confusing situations where the telepresence log file shows a kubectl command failing whereas the same command typed at the shell completes successfully. Other (worse) failure modes are possible too.

The SSH clients and server are configured to close connections after 3 seconds... this is probably too low. Increase to 10 or 30 or something.

Some people have been confused about how Telepresence works, and think it's too magical. So a few lines of explanation ("ssh tunnels with agents at both ends") would go a ways.

Let's say you want to connect to two cloud PostgreSQL databases via Telepresence, so you do:

telepresence --proxy mydb1.cloud:5432 --proxy mydb2.cloud:5432 myservice

This will fail because at the moment 5432 can only be proxied once. This limitation should be removed.

Telepresence (in 0.22 or later) cannot proxy local services running in a Docker container.

This was implemented in version < 0.22, but was not very robust. Having both local process and local container supported in same program also impacts the design in ways that aren't. The feature was therefore removed, but should be reintroduced if users want it, perhaps as a separate program.

Requirements:

• Ability to hard code destination port of a Service in the code written by the developer. E.g. "connect to port 6719 of service myredis" should work if the destination Service is on port 6719. Previously this was not possible, since we proxied ports to random other ports, so e.g. you would have to connect to port 2000.

Implementation ideas:

• Document how to run Telepresence inside a Docker container. This is the easiest option, but require more user work.
• OpenVPN or some other VPN.
• Similar to original design, except instead of having single container proxy everything, have a different container proxying each Service. That would allow them to have different IP/port combinations, much like real Services, which is necessary if we want to meet requirements above.

What were you trying to do?

I was trying to start a shiny new telepresence shell on my Mac...

What did you expect to happen?

I was expecting a shiny new telepresence shell to be running on my Mac...

What happened instead?

...but it didn't work. :(

Automatically included information

Command line: ['/Users/flynn/bin/telepresence', '--new-deployment', 'skunkworks', '--expose', '5000', '--run-shell']
Version: 0.14
Python version: 3.6.0 (default, Jan 25 2017, 07:30:03) [GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)]
kubectl version: (error: Command '['kubectl', 'version', '--short', '--client']' returned non-zero exit status 1.)
Docker version: `Client:
Version: 17.03.0-ce
API version: 1.26
Go version: go1.7.5
Git commit: 60ccb22
Built: Thu Feb 23 10:40:59 2017
OS/Arch: darwin/amd64

Server:
Version: 17.03.0-ce
API version: 1.26 (minimum version 1.12)
Go version: go1.7.5
Git commit: 3a232c8
Built: Tue Feb 28 07:52:04 2017
OS/Arch: linux/amd64
Experimental: trueOS:Darwin sasami-2.local 16.4.0 Darwin Kernel Version 16.4.0: Thu Dec 22 22:53:21 PST 2016; root:xnu-3789.41.3~3/RELEASE_X86_64 x86_64`
Traceback:

Traceback (most recent call last):
  File "/Users/flynn/bin/telepresence", line 401, in call_f
    return f(*args, **kwargs)
  File "/Users/flynn/bin/telepresence", line 505, in go
    container_id,
  File "/Users/flynn/bin/telepresence", line 292, in run_local_command
    tor_conffile.write(TORSOCKS_CONFIG.format(socks_port))
  File "/Users/flynn/.pyenv/versions/3.6.0/lib/python3.6/tempfile.py", line 483, in func_wrapper
    return func(*args, **kwargs)
TypeError: a bytes-like object is required, not 'str'

Logs:

Running: (['kubectl', 'delete', '--ignore-not-found', 'service,deployment', 'skunkworks'],)
Running: (['kubectl', 'run', '--generator', 'deployment/v1beta1', 'skunkworks', '--image=datawire/telepresence-k8s:0.14', '--port=5000', '--expose'],)
Running: (['docker', 'run', '--rm', '--name', 'telepresence-1490041216-377813-51396', '-t', '-v', '/Users/flynn:/opt:ro', '-v', '/Users/flynn:/Users/flynn:ro', '-p', ':9050', '-v', '/tmp/tmpg6z9hlk9:/output', 'datawire/telepresence-local:0.14', '502', 'skunkworks', '5000', '', '10.12.8.60'],)
Unable to find image 'datawire/telepresence-local:0.14' locally
0.14: Pulling from datawire/telepresence-local
0a8490d0dfd3: Already exists
51f9c8ec365f: Already exists
5d12fe2d195a: Pulling fs layer
74f8f9ff85ef: Pulling fs layer
5d12fe2d195a: Verifying Checksum
5d12fe2d195a: Download complete
74f8f9ff85ef: Download complete
5d12fe2d195a: Pull complete
74f8f9ff85ef: Pull complete
Digest: sha256:449e1c8ade09e57b8c97d16dcd9cdd19bb2a14da2e8d2da253125afef2042ccd
Status: Downloaded newer image for datawire/telepresence-local:0.14
Expected metadata for pods: {'labels': {'run': 'skunkworks'}, 'creationTimestamp': None}
Checking {'pod-template-hash': '3445746570', 'service': 'edge-envoy'} (phase Running)...
Labels don't match.
Checking {'pod-template-hash': '498312721', 'service': 'envoy-sds'} (phase Running)...
Labels don't match.
Checking {'pod-template-hash': '592758786', 'service': 'gruesvc'} (phase Running)...
Labels don't match.
Checking {'pod-template-hash': '1385931004', 'service': 'postgres'} (phase Running)...
Labels don't match.
Checking {'pod-template-hash': '642107786', 'run': 'skunkworks'} (phase Pending)...
Looks like we've found our pod!
Forwarding from 127.0.0.1:22 -> 22
Forwarding from [::1]:22 -> 22
Handling connection for 22
Handling connection for 22
Handling connection for 22
Forwarding from 127.0.0.1:2000 -> 2000
Forwarding from [::1]:2000 -> 2000
Forwarding from 127.0.0.1:2002 -> 2002
Forwarding from [::1]:2002 -> 2002
Forwarding from 127.0.0.1:2006 -> 2006
Forwarding from [::1]:2006 -> 2006
Forwarding from 127.0.0.1:2004 -> 2004
Forwarding from [::1]:2004 -> 2004
Forwarding from 127.0.0.1:2005 -> 2005
Forwarding from [::1]:2005 -> 2005
Forwarding from 127.0.0.1:2001 -> 2001
Forwarding from [::1]:2001 -> 2001
Forwarding fr

1. Environment variables are generated based on default namespace
2. --new-deployment creates in default namespace
3. --deployment presumes the Deployment is in the default namespace

Possibly other assumptions elsewhere.

Originally reported by @exarkun .

--proxy does not currently have an end-to-end test.

Given an existing Deployment it should be possible to extract environment variables and the like for the Telepresence Deployment.

It would be good to support Windows.

Options:

• Windows Subsystem for Linux.
• torsocks equivalent.
• Other implementation strategies, e.g. #76.

Right now we have file you download.

Some issues:

1. We don't notice new versions exist.
2. No update mechanism.
3. No way to use 3rd party Python libraries.

Options (non-exclusive), each of which might cover different subset of the above:

• Native packages (brew, deb, rpm)
• arx/nix-bundle/appimage/PyInstaller
• pex
• Rewrite in Go

Native packages seem best. Some thoughts on tooling:

• For Homebrew just make a brew/cask/tap whatever it's called. It's not hard. Maybe copy Flocker's (reverse patch of https://github.com/ClusterHQ/flocker/pull/2755/files includes most of it I think?).
• fpm makes building deb/rpm packages nice.
• https://github.com/alanfranz/fpm-within-docker wraps fpm in OS-specific docker images to build per-distro packages.
• https://github.com/krobertson/deb-s3 can make apt repo on s3.
• Probably something equally reasonable can be found for RPMs.
• https://kevinlaw.info/blog/distributing-flask-as-a-package.html explains how to combine Travis-CI and Bintray.

Based on comments in #18, it seems error handling needs some work:

1. Capture stderr and perhaps stdout from relevant subprocesses, so it doesn't get lost and can be reported automatically along with the traceback.
2. We should record the Docker version and kubectl version in the bug report.

The get_pod_name(deployment_name) logic can fail to choose the correct pod, resulting in an attempt to access a dead pod.

Currently telepresence requires kubectl and corresponding credentials to access the remote Kubernetes cluster. It would be nice if there was another access control mechanism that didn't require giving users full Kubernetes access.

Interested in this feature? Add a "thumbs up" or comment below.

I don't really know what's going on here:

$ X="-it --rm leastauthority/subscription-converger /app/env/bin/python -c xxx"
$ docker run $X
Traceback (most recent call last):
  File "<string>", line 1, in <module>
NameError: name 'xxx' is not defined
$ ../telepresence --deployment s4-infrastructure --docker-run $X
Starting proxy...
Starting local container...
ImportError: No module named site
Shutting proxy down...
$