Failure to automatically insert into the CAS about bazel-buildfarm HOT 7 CLOSED

The 1300 individual outputs here is perhaps excessive, but I don't have a stick to enforce/measure it with from the API - I could see several issues here:

• The reaction to generating a larger-than-maxMessageSize message for transport should be more resolute - a throw, with perhaps some fallback (as indicated, turning off some of the inline content).
• All of our 'repeated' entries probably bear some max counts, messages with multiple repeated entries need to be summed into 'total counts' (Directory)
• Anything where the result of fitting things into the maxMessageSize (I think it's 4MiB, but I can set it higher/make it configurable?) means that we could not act on certain requests needs to be switched to stream/chunk.
• It would be really nice if the StatusRuntimeException status for the inbound message size being exceeded in grpc was not 'CANCELLED' and told us what was going wrong here...

@ola-rozenfeld any thoughts here?

from bazel-buildfarm.

I think configuring maxMessageSize will not solve your general problem, which is -- you can't just keep inlining small outputs into ActionResult if there is a large number of such outputs. Instead of capping individual file size, cap the overall inline size. Unfortunately, you can't directly access io.grpc.internal.GrpcUtil.DEFAULT_MAX_MESSAGE_SIZE, but it is indeed 4MB.
API-wise, I think we also have to cap the overall supported number of individual outputs for an action. I would say "let's add a paging option", but that depends on how hard it is for users to formulate actions as having tree outputs instead. If it's hard, then, yeah, we should add paging options.

from bazel-buildfarm.

users to formulate actions as having tree outputs instead

What's a 'tree output'? The (pending 0.10) output_directories population and output fetch?

from bazel-buildfarm.

Yes.

from bazel-buildfarm.

Now that we've got this reducing the overall max size of the inline content, I want to break this out into a couple of tasks to make this only a remote ex API limitation (breaking from using its messages to try to store these results). We will add some testing as well...

from bazel-buildfarm.

@AustinSchuh would you mind letting us know if this is still completely broken for some of these actions? I'll wait for a week, but I think we've gotten down to scales (and inlining is still in flux) that make this work fine for now.

from bazel-buildfarm.

This guy is ancient, and if individual issues associated with size limits (like something we just found with directory hierarchy depth) come up, we can deal with them separately. Closing.

from bazel-buildfarm.