Support Parallelism about nes-py HOT 5 CLOSED

Hello! I was trying to run multiple gym-super-mario-bros environments in separate threads today, and it didn't work (Lots of "invalid OPcode"). Apparently, this is something you have been thinking about. I hope you will try to fix this. For my use case, multiple processes may be an option too, but my first choice was threading. Running multiple instances in a single thread is less interesting, as the main point is to use the power of multiple CPU cores. I am intending to use this for more efficient sampling of observations received through play with a given policy. I sample observations only with a low probability (dependent on the accumulated reward), so right now I believe the sampling is bound by the emulator, not by the harddrive. (And definitely not by the memory, if/when I move all operations into RAM).

Update: I made my frame sampling work with multiprocessing (Pool and map). This was with a rudimentary stochastic policy, not with a deep learning-based policy. I don't know if running multiple tensorflow-gpu instances in independent processes is even possible. It would be better to share one instance, as they will be running the same policy. I guess sharing a tensorflow instance is much easier using threads than using processes.

from nes-py.

Sharing data will certainly be much easier with threads opposed to processes; however, this is the current issue with the C++ codebase here that uses a functional decomposition instead of a proper object oriented design. I've attempted to refactor, but the change is nontrivial and has eaten quite a lot of my time as the C++ is not my code to begin with.

TensorFlow can run in multiple processes, but you'll need an explicitly dedicated GPU for each process or else to just use the CPU. I use this config to train multiple models (independently) on a single node equipped with multiple GPUs using the linux tool screen. To share a policy, you'd need some form of synchronization mechanism between the processes. I've seen Redis used by OpenAI to handle this sort of issue in their work, though this is obviously for a massively parallel system running across hundreds of CPU cores. Alternatively, you can probably use TF callbacks and some form of mutex/semaphore.

Ultimately, I'd suggest a multi-processing approach as python threads are a rather limited form of parallelism. However, feel free to dig into the C++ to refactor out the poor data management and support threads! A change like that would likely solve the remaining bugs in this codebase, i.e., #36

from nes-py.

from nes-py.

I feel the same way!

from nes-py.

Parallelism of all forms is now supported by the master branch. Code will release under version 3.0.0 sometime today. Closing issue

from nes-py.