Comments (12)
RayJob works best for running ephemeral clusters that are cleaned up after running a single job.
However, if you want to run additional jobs on the same RayCluster used for a RayJob, you can just use Ray CLI to submit new jobs:
ray job submit --address <address of the RayCluster from RayJob> --working-dir . -- python myjob.py
from kuberay.
In our ML platform, we want to enable users to run their jobs using our CLI. On the backend, we want to leverage the RayJob manifest to create the Ray cluster for them and then run their job on it.
@aybidi if you're users are looking to submit adhoc jobs to a cluster, I think you're better off creating RayClusters for them instead of using RayJob.
from kuberay.
Our use case is also ephemeral. We want the users to run a command like $ <our-cli> submit job
and, in the backend, (i) create a cluster, (ii) run the job, and (iii) delete the cluster.
Right now, we achieve it using RayCluster manifest (to create the cluster) and JobSubmissionClient
to submit the job to it. We want to, however, replace the JobSubmissionClient
and switch to using the RayJob
manifest.
from kuberay.
Also please let me know if you need more context -- happy to discuss it
from kuberay.
Right now, we achieve it using RayCluster manifest (to create the cluster) and JobSubmissionClient to submit the job to it. We want to, however, replace the JobSubmissionClient and switch to using the RayJob manifest.
Ah, gotcha -- sorry for misunderstanding.
It sounds like you just need to convert the $ <our-cli> submit job
call into a RayJob. The rayClusterSpec
field would be the same as the existing RayCluster you're using. You'll need to update the RayJob entrypoint field to whatever the user specified in $ <our-cli> submit job
. Hope that helps answer your question
from kuberay.
Thanks for the reply! We have figured out that bit (creating a cluster and setting the entrypoint).
We're struggling to understand how users can upload their local working dir to the Ray cluster created. Using JobSubmissionClient
, the working_dir
in the runtime_env
refers to the local directory and gets uploaded to the head node. However, when we use RayJob manifest with the runtimeEnvYAML
like so:
apiVersion: ray.io/v1
kind: RayJob
metadata:
name: rayjob-sample
spec:
entrypoint: python /home/ray/samples/sample_code.py
runtimeEnvYAML: |
working_dir: ...
it expects the working_dir
to be an existing directory in the cluster (head node's main container).
from kuberay.
it expects the working_dir to be an existing directory in the cluster (head node's main container).
I don't think this is supported. RayJob assumes the source code is available inside the container. It can reference remote zip files though (example), so maybe you can add some logic in your CLI to upload the local working directory and reference it in the RayJob? @kevin85421 do you know if there's any other workaround?
from kuberay.
providing some more context:
So if a user has a bunch of scripts in their local dev environment:
configs/
|---- custom-config.yaml
src/
|---- train.py
|---- process.py
|---- eval.py
|---- main.py
and they want to run $ <our-cli> submit job --config configs/custom-config.yaml -- python src/main.py
, we want to prepare the RayJob CR for them using the custom-config.yaml
and then submit the RayJob CR. The issue here is that the src/
files may not exist on the Ray cluster that will be created. This wasn't an issue using JobSubmissionClient
because the client uploads the src/
to the head node using an HTTP request.
from kuberay.
Online examples in your samples suggest creating a ConfigMap
object to store the scripts and then mounting it onto the container. We can create multiple key-value pairs (one for each file name and its content) in a single ConfigMap.
from kuberay.
Online examples in your samples suggest creating a ConfigMap object to store the scripts and then mounting it onto the container. We can create multiple key-value pairs (one for each file name and its content) in a single ConfigMap.
The ConfigMap was just used as an example but I don't think we would recommend this for real world use-cases (@kevin85421 correct me if I'm wrong). As soon as you need to reference multiple files / dependencies it wouldn't scale well.
I think your best bet is to add logic in your internal tool to upload remote zip of the working directory or upload a new container image that includes your local change.
from kuberay.
I see in the rayjob entrypoint, user could clone a project and run it. But I am not sure if there's GitHub Enterprise integration with Rayjob to make Rayjob easier to use. @andrewsykim , do you know if it's supported?
from kuberay.
Related Issues (20)
- [Bug] Fail the job, if the head node crashes HOT 2
- [Bug] Image vulnerabilities found with Aquasec HOT 4
- [Bug] [API Server] JobSubmission service does not work for cluster names >41 characters HOT 1
- [Bug] Update Readme to point to 1.1.1 instead of 1.1.0 for the operator HOT 1
- [Umbrella] Ray Autoscaling tests HOT 4
- [Bug] RayCluster Helm Chart: containerEnv set to null when not values are specified HOT 1
- [Umbrella] RayService HA tests
- [Bug] RayCluster Sporadic RPC Error HOT 1
- [Bug] RayJob should surface errors with underlying RayCluster HOT 7
- [Bug] "enable-batch-scheduler" bool flag is not working for schedulers other than Volcano
- [Bug] Minimum CPU and Memory requirements for KubeRay Head and worker pods
- [Feature] Allow setting `ttlSecondsAfterFinished` for RayJob submitter Job HOT 2
- [Bug] KubeRay cluster resource status is reporting Ready when there are pods still pending HOT 6
- [Feature] Display reconcile failures as events on ray clusters
- [Feature] Create an example doc for Modin HOT 1
- [Doc] Release schedule
- [Bug] Unable to open a custom application port HOT 7
- [Feature] Rayjob GitHub Enterprise integration HOT 3
- [Feature] can RayJob run in a local cluster ? HOT 6
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from kuberay.