Tasks assigned to LocalWorkerPool are not load balanced about ml-compiler-opt HOT 4 CLOSED

Kind of; the local collector isn't meant to be an efficient solution: the main constraint is the # of CPUs, and if the workload is too large, the solution is going distributed (wip). Since the overall parameters of a training setup are known - i.e. #distributed machines, #cores per machine, #modules - the naive solution may be "good enough", if the training setup is appropriately dimensioned: all compilations happen simultaneously and no module is starved.

But good to keep the issue, maybe we hit training cases when this needs addressing!

from ml-compiler-opt.

didn't isolate the specific worker no., but when I was running the validation data collector (wip)(which effectively compiles everything) it was noticeably not scheduling properly, with and without sorted-by-size modules. Might be that as Yundi pointed out in #77, early worker numbers get more work on average since it goes 12345 12345.

This is indeed mostly for the non-distributed case though

from ml-compiler-opt.

aa! I think the validation data collector should "look" more like generate_default_trace though

from ml-compiler-opt.

(capturing offline discussion) @Northbadge convinced me we can reasonably use the same distribution mechanism and abstraction (the workers/worker pool) for both - thanks!

from ml-compiler-opt.