Git Product home page Git Product logo

Comments (4)

andrewsykim avatar andrewsykim commented on August 26, 2024

@HCharlie you can interact with KubeRay created Ray clusters using Jupyter, the only requirement is that the notebook can reach the Ray head.

Here's one very basic example notebook that connects to the Ray head using Kubernetes Service domain: https://github.com/GoogleCloudPlatform/ai-on-gke/blob/main/ray-on-gke/examples/notebooks/ray_basic.ipynb

from kuberay.

HCharlie avatar HCharlie commented on August 26, 2024

@andrewsykim , thanks for the answer, this looks quite promising to me. However, I tried to portforward the service to local
kubectl port-forward service/xxxx-head-svc 8265:8265 and access it inside the notebook cell via ray.init("localhost:8265"), I see the following error after a few minutes, do you know what could go wrong?

2024-07-03 17:07:49,676	INFO worker.py:1586 -- Connecting to existing Ray cluster at address: 127.0.0.1:8265...
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
File ~/Documents/test/ray_fashion_mnist_example/venv/lib/python3.11/site-packages/ray/_private/worker.py:1727, in init(address, num_cpus, num_gpus, resources, labels, object_store_memory, local_mode, ignore_reinit_error, include_dashboard, dashboard_host, dashboard_port, job_config, configure_logging, logging_level, logging_format, logging_config, log_to_driver, namespace, runtime_env, storage, **kwargs)
   1726 try:
-> 1727     _global_node = ray._private.node.Node(
   1728         ray_params,
   1729         head=False,
   1730         shutdown_at_exit=False,
   1731         spawn_reaper=False,
   1732         connect_only=True,
   1733     )
   1734 except (ConnectionError, RuntimeError):

File ~/Documents/test/ray_fashion_mnist_example/venv/lib/python3.11/site-packages/ray/_private/node.py:159, in Node.__init__(self, ray_params, head, shutdown_at_exit, spawn_reaper, connect_only, default_worker, ray_init_cluster)
    158     self.validate_ip_port(self.address)
--> 159     self._init_gcs_client()
    161 # Register the temp dir.

File ~/Documents/test/ray_fashion_mnist_example/venv/lib/python3.11/site-packages/ray/_private/node.py:765, in Node._init_gcs_client(self)
    764     else:
--> 765         raise RuntimeError(
    766             f"Failed to {'start' if self.head else 'connect to'} GCS."
    767         )
    769 ray.experimental.internal_kv._initialize_internal_kv(self._gcs_client)

RuntimeError: Failed to connect to GCS.

During handling of the above exception, another exception occurred:

ConnectionError                           Traceback (most recent call last)
Cell In[6], line 1
----> 1 ray.init("localhost:8265")

File ~/Documents/test/ray_fashion_mnist_example/venv/lib/python3.11/site-packages/ray/_private/client_mode_hook.py:103, in client_mode_hook.<locals>.wrapper(*args, **kwargs)
    101     if func.__name__ != "init" or is_client_mode_enabled_by_default:
    102         return getattr(ray, func.__name__)(*args, **kwargs)
--> 103 return func(*args, **kwargs)

File ~/Documents/test/ray_fashion_mnist_example/venv/lib/python3.11/site-packages/ray/_private/worker.py:1743, in init(address, num_cpus, num_gpus, resources, labels, object_store_memory, local_mode, ignore_reinit_error, include_dashboard, dashboard_host, dashboard_port, job_config, configure_logging, logging_level, logging_format, logging_config, log_to_driver, namespace, runtime_env, storage, **kwargs)
   1735         if gcs_address == ray._private.utils.read_ray_address(_temp_dir):
   1736             logger.info(
   1737                 "Failed to connect to the default Ray cluster address at "
   1738                 f"{gcs_address}. This is most likely due to a previous Ray "
   (...)
   1741                 "`ray start`."
   1742             )
-> 1743         raise ConnectionError
   1745 # Log a message to find the Ray address that we connected to and the
   1746 # dashboard URL.
   1747 if ray_constants.RAY_OVERRIDE_DASHBOARD_URL in os.environ:

ConnectionError: 

from kuberay.

andrewsykim avatar andrewsykim commented on August 26, 2024

I think you want to use port 10001

kubectl port-forward service/xxxx-head-svc 10001:10001
ray.init("ray://localhost:10001")

from kuberay.

HCharlie avatar HCharlie commented on August 26, 2024

Ah, I missed this link, https://docs.ray.io/en/latest/cluster/kubernetes/user-guides/config.html#services-and-networking , now it works, thanks a lot for your fast response and help!

from kuberay.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.