opt-dist: separate dist-build from profiling about rust HOT 4 OPEN

Separate each stage and be able to start from any stage

We need to remember here that the profiles are not small and very numerous. This could quickly eat a lot of available disk space on some builders. Some of them already don't use separate profile files for space reasons. This could likely need to be opt-in, as the incremental opt-dist use-case is less suited for CI than local work.

from rust.

cc @Kobzol

from rust.

Hi, this is a good idea, although as so very often, performance considerations are in the way of clean software design :) Ideally, we'd have each stage as a separate function and just expose the stages so that they can be mixed & matched by users as they wish. However, what we have currently is that all stages are super tightly integrated, with very careful ordering and bootstrap argument selection that was hand-crafted so that the build takes as little time as possible (to be clear: as little time as possible if you want to run all the steps in one go).

I suppose that we could introduce separate commands for the individual stages, but I think that we'll need to duplicate some of the stage building code in main.rs, I'm not sure if we can make the current super tightly integrated execute_pipeline composable. But it might be worth a try :)

from rust.

We need to remember here that the profiles are not small and very numerous. This could quickly eat a lot of available disk space on some builders. Some of them already don't use separate profile files for space reasons. This could likely need to be opt-in, as the incremental opt-dist use-case is less suited for CI than local work.

After postprocessing, each profile is actually just a single file that has like 100 MiB, so this shouldn't be a concern I think. (also, I don't want to separate the stages on CI 😄)

from rust.