Proposal: Improved Handling of Parallel Processes about vivarium-core HOT 3 OPEN

I like the proposed "Process Commands" -- agreed with Ryan that this will help beyond simplifying parallel processes in that it declares a general protocol. I also find invoke_process confusing, and have to trace through the code every time.

It looks like you have worked through some of the listed downsides on the draft PR? ParallelProcess is now inheriting from Process? This seems correct.

That would be great to keep ParallelProcesses in the store hierarchy, and in Engine.processes -- a nice simplification that will save a lot of confusion. You can remove Engine.parallel without breaking any of the projects that I have helped with, which is most of them. So this won't be a problem.

So yes, this proposal looks good!

from vivarium-core.

Hey @U8NWXD, this is a great proposal. The existing parallel implementation was really just the first step I had in mind and more an expedient for getting large(r) colonies, I'm glad you're revisiting this.

I think the key you mention is making a kind of "process protocol", what you are calling Process Commands. This leads into supporting a general rpc-like api for querying and interacting with processes which could be the basis for a distributed api where each process (or engine) can run on it's own machine, coordinating through this api.

I think the protocol (let's call it a protocol) would involve the functions you mention, and in addition some more members of the Process interface, namely:

• calculate_timestep
• ports_schema
• next_update (currently the only one supported)
• end (you called it halt, it is called end in Engine.... either way)
• inspect (retrieve any relevant internal state report)

Maybe others? I think supporting __init__ would be great too, but that seems more like an Engine method which would have the ability to "allocate" new process instances.

I've already been plotting a bit along these lines so could be worthwhile discussing IRL. Or we can hash it all out here and in your other PR.

from vivarium-core.

I have a rough implementation in #198 that passes our existing tests. I still need to do some more comprehensive testing though. I decided to support all of the existing Process methods in the process commands protocol, which lets us make ParallelProcess inherit from Process. I think that will make the implementation in Engine cleaner: there will be lots of Process instances in the store hierarchy. Some of them will happen to be ParallelProcess instances, but Engine will, for the most part, be able to treat them as normal Process objects

from vivarium-core.