overv / outrun Goto Github PK

When i outrun, this happens:

user:~/ $ outrun
Traceback (most recent call last):
  File "/home/user/.local/bin/outrun", line 5, in <module>
    from outrun.__main__ import main
  File "/home/user/.local/lib/python3.11/site-packages/outrun/__main__.py", line 21, in <module>
    import outrun.operations as operations
  File "/home/user/.local/lib/python3.11/site-packages/outrun/operations/__init__.py", line 48, in <module>
    from .remote import RemoteOperations
  File "/home/user/.local/lib/python3.11/site-packages/outrun/operations/remote.py", line 15, in <module>
    from outrun.config import Config
  File "/home/user/.local/lib/python3.11/site-packages/outrun/config.py", line 32, in <module>
    @dataclass
     ^^^^^^^^^
  File "/usr/lib64/python3.11/dataclasses.py", line 1220, in dataclass
    return wrap(cls)
           ^^^^^^^^^
  File "/usr/lib64/python3.11/dataclasses.py", line 1210, in wrap
    return _process_class(cls, init, repr, eq, order, unsafe_hash,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/dataclasses.py", line 958, in _process_class
    cls_fields.append(_get_field(cls, name, type, kw_only))
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/dataclasses.py", line 815, in _get_field
    raise ValueError(f'mutable default {type(f.default)} for field '
ValueError: mutable default <class 'outrun.config.CacheConfig'> for field cache is not allowed: use default_factory

Python version is 3.11, although same error happens with 3.7 too.

FUSE 3.x is used primarily because of its support for performance enhancing features like kernel write-back caching, but version 2.x (libfuse2) is far more likely to be installed by default. Because it's the only non-Python dependency I think it would be nice if outrun can support falling back to 2.x to simplify the installation.

I usually use S3 to cache. Syntax to hit a URI for a binary or file instead of localpath? Syntax sugar to sign temp S3 files so you can execute on an S3 unprivileged remote.

Thanks for this useful project!

In your README file, you write:

You may wonder why we’re not using an existing network file system solution like NFS or SSHFS. The trouble with these is that it’s difficult to automate quick setup for ad-hoc sessions. NFS needs to be set up with configuration files and SSHFS requires the remote machine to be able to SSH back to the local machine. In contrast, the included RPC file system is a lightweight TCP server with a per-session token that can be securely tunneled over the SSH session that we're already using. Having a custom solution also opens up opportunities for a lot of optimizations as we'll see later on.

I would like to make you aware of an elegant way to use SSHFS: please see the paragraph “reverse sshfs” in my article https://michael.stapelberg.ch/posts/2018-03-13-cpu/, where I describe that one can start sftp-server and connect it with an sshfs process running on the remote machine via SSH input/output redirection.

Perhaps you can switch to that and get rid of the custom FUSE file system, if it compares favorably performance-wise.

The readme doesn't mention it, but /etc/machine-id needs to be present otherwise you get a mysterious error:

ERROR - failed to run command: root file system mount failed: [Errno 2] No such file or directory

Dear outrun dev team,

Just bumped into this awesome tool and I'd like to send kudos to the dev team for implementing it.
I was wondering: would it be imaginable to allow outrun to be used as a Python decorator?
Imagine I'm working in a Jupyter Notebook in one cloud machine that is very small but ok for notebooking. Now imagine that one of my analysis Python (or wrapped in Python code) functions requires a significant amount of computing power that is not available on the machine I'm working on.
In my work environment, it's very simple for me to fire-up a more powerful machine, but this requires me to migrate my notebook to this new machine. However, if I could just decorate my computing function in my notebook indicating that it has to run in another machine, this would make my life 1000x easier.

It would be extremely helpful if outrun could support this use case. How does it sound?

Cheers,

Hugo

Example of current behavior:

$ outrun njodell@outrun-vm whoami
fuse: warning: library too old, some operations may not not work
root

I'd like the program I'm running to execute as the same user that I logged in as.

It may be possible to run cross architecture binaries using qemu-user.

Even though this means emulation, it would be helpful since ARM machines are mostly slower currently than server-class x86 machines.

It would be nice if outrun could install itself on the targeted remote machine if the right version is not installed yet. Ideally it can do this without requiring pip to be available, because it's often not installed by default.

My desktop setup is running POP OS 20.04

I installed outrun and libfuse on a linode nanode running ubuntu server 20.04.1 , linux kernel 5.4.0-42
yet, for some reason, it's still not working

will@[redacted]:/Desktop$ outrun --debug $nanode1 python3 outrunTest.py outText 200 5
2020-08-29 03:33:41,821 - DEBUG - running ['ssh', '-o', 'LogLevel=error', '-R', '30920:localhost:30920', '-R', '30581:localhost:30581', '-R', '31159:localhost:31159', '-tt', '[redacted]', 'outrun', '--remote', '--unshare', '--protocol=1.0.0', '--platform=x86_64', '--config=/.outrun/config', '--timeout=5000', '--environment-port=30920', '--filesystem-port=30581', '--debug', '--cache-port=31159', '.', '.']
2020-08-29 07:33:44,488 - INFO - no config file at /root/.outrun/config
2020-08-29 07:33:45,426 - DEBUG - rpc::None() - 932 ms
2020-08-29 07:33:45,427 - ERROR - failed to run command: root file system mount failed: token mismatch between client and server

Hi,

First thanks for implementing such an amazing idea, I love it.
I'm playing around with outrun and would like to pass extra options to ssh command.

when I use outrun --ssh '-vvv' user@host ls, it says :

usage: outrun [option...] destination command [arg...]
outrun: error: argument --ssh: expected one argument

when I use outrun --ssh 'ssh -vvv' user@host ls, it says :

2021-06-03 10:25:24,381 - ERROR - failed to run command: ssh failed: OpenSSH_8.4p1 Debian-5, OpenSSL 1.1.1k  25 Mar 2021
debug1: Reading configuration data /home/user/.ssh/config
debug1: /home/user/.ssh/config line 2: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/user/.ssh/known_hosts'
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/user/.ssh/known_hosts2'
debug2: resolving "ssh" port 22
ssh: Could not resolve hostname ssh: Name or service not know

When using debug, I get that :

2021-06-03 10:31:13,540 - DEBUG - running ['ssh', '-o', 'LogLevel=error', '-R', '31138:localhost:31138', '-R', '31828:localhost:31828', '-R', '31034:localhost:31034', '-tt', 'ssh', '-vvv', '[email protected]', 'outrun', '--remote', '--unshare', '--protocol=1.0.0', '--platform=x86_64', '--config=~/.outrun/config', '--timeout=5000', '--environment-port=31138', '--filesystem-port=31828', '--debug', '--cache-port=31034', '.', '.']

which suggest that the first syntax is the right one.

It seems that --ssh requires at least two values in quotes.

It seems to come from there but I've not been able to fix it.

It can be workaround by using --ssh '-v -v -v' for example but it is not immediate.

I'd like to use outrun for a fairly complex task, where I want to ensure that IO across the local/remote boundary isn't a bottleneck (and where I'm not sure where and whether a lot of that kind of IO might be happening).

My thought in evaluating outrun in this context was to modify outrun to log this proxied IO to a file, via a new optional flag. So I'm wondering:

• is this a good idea, or is there some trivial way to do this externally to outrun (I suppose if I could log the amount of data going over the ssh connection that would be sufficient... but wouldn't support debugging)
• any thoughts on what sort of design you'd consider merging if I made a PR?

Thanks! This is a really cool project

It would be nice if outrun could store its files in the standard locations.

• The outrun config would be stored in $XDG_CONFIG_HOME so the config file would typically be ~/.config/outrun.conf.
• The outrun cache would be stored in $XDG_CACHE_HOME so the cache would typically be ~/.cache/outrun.

References:

• Arch Wiki entry
• Freedesktop specification

Hello,

I am trying to start outrun but get the following error. How can I get more details about what's happening?

xxx@pop-os:~/.local/lib/python3.8/site-packages/outrun$ outrun user@xxx ls
Password:
Verification code:
bash: outrun: command not found
2020-08-14 22:26:39,767 - ERROR - failed to run command: remote outrun failed to start

thanks
Siddharth

ie some cases command or packages need be installed.
ie Distro = Gentoo emerge -vgk
(verbose get pkgs use pkgs if available else build) or yum/apt etc..

iot nodes arm64 >>>> amd-64-server to do bulk processing going to have a bad day..
ie network monitoring nodes or honeypot nodes ... else use cmd only on the server node ie ship data.. for crunching.
without having to wrap qemu and ie use same util on host/s could ignore arch requirements and just do the data.

however if just data files then good times.. as if one has to wrapp the cmd's ie qumeu etc ie fchroot it could add layers of unexpected complexity.

else for builds fchroot might be of use as a sub.. /optional util....
( https://pypi.org/project/fchroot/ / https://code.funtoo.org/bitbucket/users/drobbins/repos/fchroot/browse )

(for Gentoo builds hopefully can add soon a common NAS-Drive for common mounts ie disfiles packages to share etc between nodes despite this tool... however emerge foo bar etc would be nice to push emerge to other nodes ie if they can build in some packages in parallel a plus, as rpi-3 rpi-4 rpi4-desktop-minilcd-case@256GB card , rock64pro , others smaller , rock64pro debian sdcard , chroot to emmc to build base Gentoo root install on. however distcc or task distribution a plus. ) however this tool cough my eye for parallel builds on multiple nodes.

adding an ebuild for this is on my to do list... as pip and gentoo dont mix terribly well, except virtual-env and $USER/./pip...../python$

It would be nice to hear if there are any very compelling usecases/workflows that people use outrun for.
I see the appeal of being able to run compute-intensive operations on a remote machine, but I feel like in most cases I just directly work on those remote machines instead of doing things locally at all.

So it would be great to see some people's popular workflows or specific usecases to see if I should be modifying my workflow for outrun.

Trying to compile some ocaml and running into this, not sure where to begin to look.

$ outrun gary@gary-dev dune build
Error: open: /tmp/dune22fe0d.output: Permission denied

Thoughts?

I don't expect this to be fast due to i/o of lots of small files, but wanted to try it :-).

You fixed a breaking bug with using default_factory. Can you release that to PyPi so its easier to install with pip?