Sample code to run DistSPECTRL.
Create new python 3.9 environment with required packages using
conda create --name distspectrl python=3.9
conda activate distspectrl
pip install -r requirements.txt
├── README.md
├── requirements.txt # requirements file installed via pip
├── train.py # training script
├── test_multi_exp_PPO.sh # bash script to run multiple experiments at once
├── gym_waysharing # (dir) 2D and 3D Navigation Environment
├── rm_cooperative_marl # (dir) MARL reward machine code from Neary et al. with modifications for Ray
├── spectrl # (dir) spectrl code provided by Jothimurugan et al. (as-is)
├── img # (dir) sample images of results, created by functions in src.visualize
└── src
├── main
│ ├── MA_monitor # Class for representing composite task monitors
│ ├── MA_spec_compiler # Composition and compilation functions
│ └── reward_shape_PPO_wrapper # Specs and Wrapped Environments
├── env_wrappers
│ ├── MA_learning # Wrapper functions for RLLib MA Environments and DistSPECTRL
│ └── MA_learning_logNscaling # Wrapper functions for MA-Dec Scaling
├── models
│ └── centralized_critic # Centralized Critic functions for CPPO mode
├── tests # (dir) simple testing scripts using unittest
└── visualize # (dir) functions to plot results
See src.main.reward_shape_PPO_wrapper.py for more example specs. From within the main directory run
conda activate distspectrl
# \phi_3
python train.py --spec_id 3 --algorithm CPPO --env navenv_inlineDS --num_workers_per_device 2 --num_cpus 8 --exp_name test --horizon 400 --train_batch_size 4800
# \phi_a
python train.py --spec_id 0 --algorithm CPPO --env nav3D_inlineDS --num_workers_per_device 2 --num_cpus 8 --exp_name test --horizon 400 --train_batch_size 4800
# \phi_3 MA-Dec Scaling, N=6 agents
python train.py --spec_id 3 --algorithm CPPO --env navenv_inlineDS_logN --num_workers_per_device 2 --num_agents 6 --num_cpus 8 --exp_name test --horizon 400 --train_batch_size 4800
# \phi_3 centralized SPECTRL, N=6 agents
python train.py --spec_id 3 --algorithm PPO --env navenv_spectrl_sa --num_workers_per_device 2 --num_agents 6 --num_cpus 8 --exp_name test --horizon 400 --train_batch_size 4800
Training template (check train.py for more options)
python train.py --spec_id <spec-id (int)> --algorithm <PPO|CPPO> --env <env_name> --num_workers_per_device 2 --num_agents <2-10 (int) > --num_cpus <NUM_CPUS> --exp_name <Experiment Name> --horizon <horizon> --train_batch_size <training batch size>
Environment name can be (nav_ | nav3D_) + <env name option>
Env. name options to choose various modes
- inlineDS - vanilla DistSPECTRL
- inlineDS_logN - MA-Dec Scaling
- inlineDS_no_monitor - no monitor state mode
- spectrl_sa - centralized SPECTRL (needs PPO algorithm)
- rmachine - our rmachine implementation based on the author's released code
Note: Our rmachine implementation uses a modified environment in rm_cooperative_marl.src.Environments.rendezvous_continuous and is only created for one spec (
We also include a bash script test_multi_exp_PPO.sh to run multiple experiments at once on the same machine.
Run python -m unittest -v src/tests/spectrl_test.py to test a few local specs in a centralized implementation.
Tools are in src/visualize Use the notebook Plot_Graphs.ipynb
Custom metrics of interest are final_state_reached_mean, max_depth_mean and stage_reached_mean.
This code was heavily built off the original SPECTRL code by Jothimurugan et al and we thank the authors for their efforts. We also thank Neary et al for their multi agent reward machine implementation.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent