Here is simple implementation of Muesli algorithm. Muesli has same performance and network architecture as MuZero, but it can be trained without MCTS lookahead search, just use one-step lookahead. It can reduce computational cost significantly compared to MuZero.

Paper : Muesli: Combining Improvements in Policy Optimization, Hessel et al., 2021 (v2 version)

This repository will be developed as part of the collaborative research with UdeM. Thanks for making this great experience and I hope this things to be useful for further progress. This codebase needs the hand of many talented contributers. Please feel free to contribute and contact!

The goal is making distributed muesli algorithm for large scale training can be intergrated with below works,

https://github.com/AGI-Collective/mini_ada

https://github.com/AGI-Collective/u3

https://github.com/Farama-Foundation/Minigrid

And we consider using https://github.com/kakaobrain/brain-agent for distributed reinforcement learning.

1. Install Docker
2. Download Dockerfile
3. Build Dockerfile docker build --build-arg git_config_name="your_git_name" --build-arg git_config_email="your_git_email" --build-arg CACHEBUST=$(date +%s) -t muesli_image .
4. Run docker image (Adjust options for your device configuration) docker run --gpus all -p 8888:8888 -p 8080:8080 -p 6006:6006 -p 6007:6007 -p 6008:6008 --name mu --rm -it muesli_image
5. Copy the jupyterlab token (If you want to make it background process, press Ctrl + P,Q)
6. Login to the jupyterlab through browser or jupyterlab desktop http://your_local_or_server_ip:8888 with token
7. Launch HPO experiment with nni (on the jupyterlab terminal) nnictl create -f --config config.yml
8. Access nni through browser http://your_local_or_server_ip:8080
9. Launch Tensorboard on the jupyterlab terminal (use one more bash terminal) tensorboard --logdir ./nni-experiments/_latest/trials --bind_all (for seeing every experiment's TB logs in one page)
10. Access Tensorboard through browser http://your_local_or_server_ip:6006

• jupyterlab-git and jupyter-collaboration are installed.
• Code was cloned into container when build, and it will be removed when container closed.
• If you want use bash shell on jupyterlab, just type ‘bash’ and press enter on the default terminal.

• You can see experiments on ‘Trials detail’ tab, and see hyperparameters by using Add/Remove columns button.
• NOTE: the hyperparameters displayed in the nni page are mismatched with experiments, so using Tensorboard HPARAMS tab is recommended.
• (log_dir of nni is changed for fixing issue about launching the TensorBoard)

• Launch TensorBoard through MS nni. Click the checkbox to the left of the trial number and click TensorBoard button.
• Or use tensorboard --logdir ./nni-experiments/_latest/trials --bind_all for check every experiments.
• About TensorBoard image slide precision
  • TensorBoard use the reservoir sampling, so some images in the episode can be skipped. If you want slide rendered images more precisely, launch TensorBoard manually by this command tensorboard --logdir . --samples_per_plugin images=100 --bind_all (directory: nni-experiments/_latest/trials/your_trial_ID/output/tensorboard) (it can be checked on terminal output)

• python -m pdb Muesli_code.py --debug

• Wiki
  • Current progress & Near-term plans
  • Model architecture
  • Model card