overv / outrun Goto Github PK
View Code? Open in Web Editor NEWExecute a local command using the processing power of another Linux machine.
License: Apache License 2.0
Execute a local command using the processing power of another Linux machine.
License: Apache License 2.0
It would be nice to hear if there are any very compelling usecases/workflows that people use outrun for.
I see the appeal of being able to run compute-intensive operations on a remote machine, but I feel like in most cases I just directly work on those remote machines instead of doing things locally at all.
So it would be great to see some people's popular workflows or specific usecases to see if I should be modifying my workflow for outrun.
It may be possible to run cross architecture binaries using qemu-user
.
Even though this means emulation, it would be helpful since ARM machines are mostly slower currently than server-class x86 machines.
I'd like to use outrun
for a fairly complex task, where I want to ensure that IO across the local/remote boundary isn't a bottleneck (and where I'm not sure where and whether a lot of that kind of IO might be happening).
My thought in evaluating outrun
in this context was to modify outrun
to log this proxied IO to a file, via a new optional flag. So I'm wondering:
outrun
(I suppose if I could log the amount of data going over the ssh connection that would be sufficient... but wouldn't support debugging)Thanks! This is a really cool project
The readme doesn't mention it, but /etc/machine-id
needs to be present otherwise you get a mysterious error:
ERROR - failed to run command: root file system mount failed: [Errno 2] No such file or directory
Example of current behavior:
$ outrun njodell@outrun-vm whoami
fuse: warning: library too old, some operations may not not work
root
I'd like the program I'm running to execute as the same user that I logged in as.
Trying to compile some ocaml and running into this, not sure where to begin to look.
$ outrun gary@gary-dev dune build
Error: open: /tmp/dune22fe0d.output: Permission denied
Thoughts?
I don't expect this to be fast due to i/o of lots of small files, but wanted to try it :-).
Hello,
I am trying to start outrun but get the following error. How can I get more details about what's happening?
xxx@pop-os:~/.local/lib/python3.8/site-packages/outrun$ outrun user@xxx ls
Password:
Verification code:
bash: outrun: command not found
2020-08-14 22:26:39,767 - ERROR - failed to run command: remote outrun failed to start
thanks
Siddharth
It would be nice if outrun could install itself on the targeted remote machine if the right version is not installed yet. Ideally it can do this without requiring pip
to be available, because it's often not installed by default.
Dear outrun dev team,
Just bumped into this awesome tool and I'd like to send kudos to the dev team for implementing it.
I was wondering: would it be imaginable to allow outrun to be used as a Python decorator?
Imagine I'm working in a Jupyter Notebook in one cloud machine that is very small but ok for notebooking. Now imagine that one of my analysis Python (or wrapped in Python code) functions requires a significant amount of computing power that is not available on the machine I'm working on.
In my work environment, it's very simple for me to fire-up a more powerful machine, but this requires me to migrate my notebook to this new machine. However, if I could just decorate my computing function in my notebook indicating that it has to run in another machine, this would make my life 1000x easier.
It would be extremely helpful if outrun could support this use case. How does it sound?
Cheers,
Hugo
Hi,
First thanks for implementing such an amazing idea, I love it.
I'm playing around with outrun and would like to pass extra options to ssh command.
when I use outrun --ssh '-vvv' user@host ls
, it says :
usage: outrun [option...] destination command [arg...]
outrun: error: argument --ssh: expected one argument
when I use outrun --ssh 'ssh -vvv' user@host ls
, it says :
2021-06-03 10:25:24,381 - ERROR - failed to run command: ssh failed: OpenSSH_8.4p1 Debian-5, OpenSSL 1.1.1k 25 Mar 2021
debug1: Reading configuration data /home/user/.ssh/config
debug1: /home/user/.ssh/config line 2: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/user/.ssh/known_hosts'
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/user/.ssh/known_hosts2'
debug2: resolving "ssh" port 22
ssh: Could not resolve hostname ssh: Name or service not know
When using debug, I get that :
2021-06-03 10:31:13,540 - DEBUG - running ['ssh', '-o', 'LogLevel=error', '-R', '31138:localhost:31138', '-R', '31828:localhost:31828', '-R', '31034:localhost:31034', '-tt', 'ssh', '-vvv', '[email protected]', 'outrun', '--remote', '--unshare', '--protocol=1.0.0', '--platform=x86_64', '--config=~/.outrun/config', '--timeout=5000', '--environment-port=31138', '--filesystem-port=31828', '--debug', '--cache-port=31034', '.', '.']
which suggest that the first syntax is the right one.
It seems that --ssh
requires at least two values in quotes.
It seems to come from there but I've not been able to fix it.
It can be workaround by using --ssh '-v -v -v'
for example but it is not immediate.
FUSE 3.x is used primarily because of its support for performance enhancing features like kernel write-back caching, but version 2.x (libfuse2
) is far more likely to be installed by default. Because it's the only non-Python dependency I think it would be nice if outrun can support falling back to 2.x to simplify the installation.
When i outrun, this happens:
user:~/ $ outrun
Traceback (most recent call last):
File "/home/user/.local/bin/outrun", line 5, in <module>
from outrun.__main__ import main
File "/home/user/.local/lib/python3.11/site-packages/outrun/__main__.py", line 21, in <module>
import outrun.operations as operations
File "/home/user/.local/lib/python3.11/site-packages/outrun/operations/__init__.py", line 48, in <module>
from .remote import RemoteOperations
File "/home/user/.local/lib/python3.11/site-packages/outrun/operations/remote.py", line 15, in <module>
from outrun.config import Config
File "/home/user/.local/lib/python3.11/site-packages/outrun/config.py", line 32, in <module>
@dataclass
^^^^^^^^^
File "/usr/lib64/python3.11/dataclasses.py", line 1220, in dataclass
return wrap(cls)
^^^^^^^^^
File "/usr/lib64/python3.11/dataclasses.py", line 1210, in wrap
return _process_class(cls, init, repr, eq, order, unsafe_hash,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/dataclasses.py", line 958, in _process_class
cls_fields.append(_get_field(cls, name, type, kw_only))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/dataclasses.py", line 815, in _get_field
raise ValueError(f'mutable default {type(f.default)} for field '
ValueError: mutable default <class 'outrun.config.CacheConfig'> for field cache is not allowed: use default_factory
Python version is 3.11, although same error happens with 3.7 too.
ie some cases command or packages need be installed.
ie Distro = Gentoo emerge -vgk
(verbose get pkgs use pkgs if available else build) or yum/apt etc..
iot nodes arm64 >>>> amd-64-server to do bulk processing going to have a bad day..
ie network monitoring nodes or honeypot nodes ... else use cmd only on the server node ie ship data.. for crunching.
without having to wrap qemu and ie use same util on host/s could ignore arch requirements and just do the data.
however if just data files then good times.. as if one has to wrapp the cmd's ie qumeu etc ie fchroot it could add layers of unexpected complexity.
else for builds fchroot might be of use as a sub.. /optional util....
( https://pypi.org/project/fchroot/ / https://code.funtoo.org/bitbucket/users/drobbins/repos/fchroot/browse )
(for Gentoo builds hopefully can add soon a common NAS-Drive for common mounts ie disfiles packages to share etc between nodes despite this tool... however emerge foo bar etc would be nice to push emerge to other nodes ie if they can build in some packages in parallel a plus, as rpi-3 rpi-4 rpi4-desktop-minilcd-case@256GB card , rock64pro , others smaller , rock64pro debian sdcard , chroot to emmc to build base Gentoo root install on. however distcc or task distribution a plus. ) however this tool cough my eye for parallel builds on multiple nodes.
adding an ebuild for this is on my to do list... as pip and gentoo dont mix terribly well, except virtual-env and
My desktop setup is running POP OS 20.04
I installed outrun and libfuse on a linode nanode running ubuntu server 20.04.1 , linux kernel 5.4.0-42
yet, for some reason, it's still not working
will@[redacted]:/Desktop$ outrun --debug $nanode1 python3 outrunTest.py outText 200 5/.outrun/config', '--timeout=5000', '--environment-port=30920', '--filesystem-port=30581', '--debug', '--cache-port=31159', '.', '.']
2020-08-29 03:33:41,821 - DEBUG - running ['ssh', '-o', 'LogLevel=error', '-R', '30920:localhost:30920', '-R', '30581:localhost:30581', '-R', '31159:localhost:31159', '-tt', '[redacted]', 'outrun', '--remote', '--unshare', '--protocol=1.0.0', '--platform=x86_64', '--config=
2020-08-29 07:33:44,488 - INFO - no config file at /root/.outrun/config
2020-08-29 07:33:45,426 - DEBUG - rpc::None() - 932 ms
2020-08-29 07:33:45,427 - ERROR - failed to run command: root file system mount failed: token mismatch between client and server
I usually use S3 to cache. Syntax to hit a URI for a binary or file instead of localpath? Syntax sugar to sign temp S3 files so you can execute on an S3 unprivileged remote.
Thanks for this useful project!
In your README file, you write:
You may wonder why we’re not using an existing network file system solution like NFS or SSHFS. The trouble with these is that it’s difficult to automate quick setup for ad-hoc sessions. NFS needs to be set up with configuration files and SSHFS requires the remote machine to be able to SSH back to the local machine. In contrast, the included RPC file system is a lightweight TCP server with a per-session token that can be securely tunneled over the SSH session that we're already using. Having a custom solution also opens up opportunities for a lot of optimizations as we'll see later on.
I would like to make you aware of an elegant way to use SSHFS: please see the paragraph “reverse sshfs” in my article https://michael.stapelberg.ch/posts/2018-03-13-cpu/, where I describe that one can start sftp-server
and connect it with an sshfs
process running on the remote machine via SSH input/output redirection.
Perhaps you can switch to that and get rid of the custom FUSE file system, if it compares favorably performance-wise.
You fixed a breaking bug with using default_factory. Can you release that to PyPi so its easier to install with pip?
It would be nice if outrun could store its files in the standard locations.
$XDG_CONFIG_HOME
so the config file would typically be ~/.config/outrun.conf
.$XDG_CACHE_HOME
so the cache would typically be ~/.cache/outrun
.References:
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.