when blob file miss，rbe client is stuck about bazel-buildfarm HOT 6 OPEN

Supplement to Question 1: Why doesn‘t verify the existence of output's blob files in the GetActionResult interface? Maybe just randomly select a worker from the cas work list to judge whether a single cas has blob file.

from bazel-buildfarm.

I've experienced this issue with autoscaling - worker would scale down and bazel client would get stuck waiting forever.

from bazel-buildfarm.

I've experienced this issue with autoscaling - worker would scale down and bazel client would get stuck waiting forever.

Hi，80degreeswest.
Are your execute worker and cas worker together （this indicates that the blob file is lost, but the redis data is not synchronized）?
Which version of bazel do you use?

I have previously tested this scenario with bazel 6.1 and it performed successfully. Of course, the android rbe we used was stuck.

In addition, I'd like to ask you how you deal with this problem.

from bazel-buildfarm.

Hi, @80degreeswest
I noticed that #976 could solve my problem, but not merge.
I tested the efficiency of adding check disk storage before and after. It doesn't seem to add much time at the moment.

from bazel-buildfarm.

Yes I use cas+execute workers. I see this problem when my workers scale down. We use bazel 5.3.1. To work around it you can enable graceful shutdown, which is available in v2.6.1. This config will wait x seconds for any executions in progress to finish before shutting down the worker. Obviously not going to help if your worker is already broken but it will solve the issue in case of normal shutdown. https://github.com/bazelbuild/bazel-buildfarm/blob/main/examples/config.yml#L128.

from bazel-buildfarm.

I'm not sure what the state of that PR is. @luxe may be able to provide some more detail on if it would make sense to re-visit it. @shirchen do you have that change deployed in your cluster?

from bazel-buildfarm.