FT GCS should handle draining of node where head pod is scheduled about kuberay HOT 3 OPEN

Hi @abatilo, thank you for opening the issue. You may have some misunderstanding for GCS FT. You can read https://docs.ray.io/en/latest/cluster/kubernetes/user-guides/kuberay-gcs-ft.html#kuberay-gcs-ft for more details. Currently, the only use case of GCS FT is for Ray Serve high availability.

from kuberay.

Thanks for responding @kevin85421. My original question still stands then. Is there no way to have the jobs running in progress survive after the Ray head gets restarted? I see that there's a way to resume a trainer from previous state but do I always have to re-submit that if the Ray head is gone?

from kuberay.

My current understanding is that Ray Train provides some degree of fault tolerance.

• If a Ray worker Pod crashes, Ray Train will launch new Ray tasks or actors, allowing the job to continue running.
• If the Ray head crashes, the driver process, which is typically running on the Ray head of the Ray job, will also crash. Therefore, the Ray tasks or actors will also be garbage collected automatically. In most cases, Ray tasks or actors are fate-sharing with the driver process. Only detached actors do not share this fate with the driver.
  • In this case, users need to write the fault tolerance logic at the application level, specifically in their Ray Python script.

    if (checkpoint exists):
      read checkpoint
    else:
      start from scratch
    
    train the model

The long-running Ray job is currently on our roadmap of the next release. We are currently working on:

• Retry mechanism in both RayJob CRD / ray job submit.
• Best practice for checkpointing.

If you are interested in this topic, you can reach out to me on Ray Slack (my handle is 'Kai-Hsun Chen (ray team)'). We can discuss your requirements and ensure there are no feature gaps for your use cases.

from kuberay.