[RayServe] Autoscaling Issue with Neuron Devices (Inf2), RayServe, and Karpenter on EKS about ray HOT 4 OPEN

@vara-bonthu Can you scope this down to just what's missing/ wrong in Ray Serve? Is this really an issue that's required to change Ray code? Or if you already know what's the fix, also feel free to contribute to the codebase

from ray.

@vara-bonthu those are great findings! If you are develop on a mac, you can try those instructions to setup it up locally https://docs.ray.io/en/master/ray-contribute/development.html#building-ray-on-linux-macos-full

If this has to go onto a cluster, I think you can raise a draft PR and one of the CI step will generate a wheel that you can use to build the docker image for testing. This is an example of such build that generates the wheel.

from ray.

Does Ray for neuron support autoscaling based on neuron devices - represented by the device plugin as aws.amazon.com/neuron ?

from ray.

@GeneDer The issue is related solely to scaling new nodes for inf2 instances. It appears node_types is not set, causing the code to break at this line in here .

What is the easiest way to debug the code to print the values passed to this method?

I will try to dig further and keep you posted

from ray.