Comments (4)
@HCharlie you can interact with KubeRay created Ray clusters using Jupyter, the only requirement is that the notebook can reach the Ray head.
Here's one very basic example notebook that connects to the Ray head using Kubernetes Service domain: https://github.com/GoogleCloudPlatform/ai-on-gke/blob/main/ray-on-gke/examples/notebooks/ray_basic.ipynb
from kuberay.
@andrewsykim , thanks for the answer, this looks quite promising to me. However, I tried to portforward the service to local
kubectl port-forward service/xxxx-head-svc 8265:8265
and access it inside the notebook cell via ray.init("localhost:8265")
, I see the following error after a few minutes, do you know what could go wrong?
2024-07-03 17:07:49,676 INFO worker.py:1586 -- Connecting to existing Ray cluster at address: 127.0.0.1:8265...
---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
File ~/Documents/test/ray_fashion_mnist_example/venv/lib/python3.11/site-packages/ray/_private/worker.py:1727, in init(address, num_cpus, num_gpus, resources, labels, object_store_memory, local_mode, ignore_reinit_error, include_dashboard, dashboard_host, dashboard_port, job_config, configure_logging, logging_level, logging_format, logging_config, log_to_driver, namespace, runtime_env, storage, **kwargs)
1726 try:
-> 1727 _global_node = ray._private.node.Node(
1728 ray_params,
1729 head=False,
1730 shutdown_at_exit=False,
1731 spawn_reaper=False,
1732 connect_only=True,
1733 )
1734 except (ConnectionError, RuntimeError):
File ~/Documents/test/ray_fashion_mnist_example/venv/lib/python3.11/site-packages/ray/_private/node.py:159, in Node.__init__(self, ray_params, head, shutdown_at_exit, spawn_reaper, connect_only, default_worker, ray_init_cluster)
158 self.validate_ip_port(self.address)
--> 159 self._init_gcs_client()
161 # Register the temp dir.
File ~/Documents/test/ray_fashion_mnist_example/venv/lib/python3.11/site-packages/ray/_private/node.py:765, in Node._init_gcs_client(self)
764 else:
--> 765 raise RuntimeError(
766 f"Failed to {'start' if self.head else 'connect to'} GCS."
767 )
769 ray.experimental.internal_kv._initialize_internal_kv(self._gcs_client)
RuntimeError: Failed to connect to GCS.
During handling of the above exception, another exception occurred:
ConnectionError Traceback (most recent call last)
Cell In[6], line 1
----> 1 ray.init("localhost:8265")
File ~/Documents/test/ray_fashion_mnist_example/venv/lib/python3.11/site-packages/ray/_private/client_mode_hook.py:103, in client_mode_hook.<locals>.wrapper(*args, **kwargs)
101 if func.__name__ != "init" or is_client_mode_enabled_by_default:
102 return getattr(ray, func.__name__)(*args, **kwargs)
--> 103 return func(*args, **kwargs)
File ~/Documents/test/ray_fashion_mnist_example/venv/lib/python3.11/site-packages/ray/_private/worker.py:1743, in init(address, num_cpus, num_gpus, resources, labels, object_store_memory, local_mode, ignore_reinit_error, include_dashboard, dashboard_host, dashboard_port, job_config, configure_logging, logging_level, logging_format, logging_config, log_to_driver, namespace, runtime_env, storage, **kwargs)
1735 if gcs_address == ray._private.utils.read_ray_address(_temp_dir):
1736 logger.info(
1737 "Failed to connect to the default Ray cluster address at "
1738 f"{gcs_address}. This is most likely due to a previous Ray "
(...)
1741 "`ray start`."
1742 )
-> 1743 raise ConnectionError
1745 # Log a message to find the Ray address that we connected to and the
1746 # dashboard URL.
1747 if ray_constants.RAY_OVERRIDE_DASHBOARD_URL in os.environ:
ConnectionError:
from kuberay.
I think you want to use port 10001
kubectl port-forward service/xxxx-head-svc 10001:10001
ray.init("ray://localhost:10001")
from kuberay.
Ah, I missed this link, https://docs.ray.io/en/latest/cluster/kubernetes/user-guides/config.html#services-and-networking , now it works, thanks a lot for your fast response and help!
from kuberay.
Related Issues (20)
- [Bug] Envtests should run specs in sequential order
- Replace references of rayproject/ray-ml image HOT 2
- What's the behavior for a Pod with restartPolicy Always when it is in PodFailed? HOT 1
- Conditions HeadPodReady and RayClusterReady have overlapping meanings HOT 4
- [Umbrella] Remove the RayCluster `.Status.State` field HOT 1
- Make sure RayService example still work
- [Umbrella] Rewrite sample YAML tests with Golang
- [Refacter] Create a new Golang package under `ray-operator/test` folder and run those tests in CI
- [Bug] RayJob can have JobStatus = Running while JobDeploymentStatus = Failed HOT 5
- [Bug] RayService Restarts After Upgrading KubeRay to v1.2.0-rc.0 HOT 11
- [Bug] RayJob submitter Job failures leads to flaky RayJobs
- [Bug] RayJob submitter Job failures leads to flaky RayJobs HOT 1
- [Request] Test Ray#46861 does not break Kuberay redis cleanup
- [Feature] I want to be able to add annotations to the ServiceAccount that RayCluster creates
- [RFC] Introduce new API-RayCluster Fleet and ReplicaSet in KubeRay HOT 3
- [Bug] Head node autoscaler container fails to communicate to the kubernetes api with a 401 in Azure Kubernetes 1.30 HOT 1
- [Feature] [API Server] head svc ingress should include rewrite target annotation as default
- [Bug] Cannot run "Serve a MobileNet image classifier on Kubernetes" doc HOT 1
- [Bug] Cannot run "Train a PyTorch model on Fashion MNIST with CPUs on Kubernetes" doc HOT 3
- [Bug] Updating min/max replicas in the RayService spec doesn't trigger the RayCluster resource update
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from kuberay.