In this project, we attempt to equip the agent with the recognition of basic components of an Atari game environment through curriculum learning—gleaned from human developmental psychology—and evaluate its performance. Our best agent was pre-trained on a carefully designed curriculum to learn to complete a new game 5x faster than regular agents. By analyzing our agent’s behavior, we can examine how it acquires knowledge and skills differently from humans.

This code is modified from https://github.com/uvipen/Street-fighter-A3C-ICM-pytorch. Specifically, the major changes are made in src/env.py. Please install PLE and the dependencies listed in requirements-dev.txt. Let me know if there is any bug. Thanks!

# running on CPUs
python train.py --num_processes 16 --map_file fire_3.txt --exp my_exp
python test.py --resume_path trained_models/my_exp/500K --map_file fire_3.txt
# visualize with tensorboard
tensorboard --logdir tensorboard --bind_all &

• I found that training speed on CPU is on par with GPU. Probably because of larger CPU memory and the on-policy training mechanism. CIMS's crunchy servers seems to be good choices.
• Not recommend using to many processes, the A3C paper uses 16 processes.
• --max_steps needs to be greater than 500 for longer exploration
• Decreasing the learning rate and changing the weight (--lambda) for curiosity loss might be worth tuning.
• The ICM paper uses --lr=1e-3. However, from my experiments it can't be that large.
• Previous papers use --frame_skip=4.
• Removing the null action (or set --num_action=5) doesn't help.
• Adding negative rewards doesn't work on map.txt. It's worth looking into how to change the hyperparameters to make it work.
• The initial position of the player can be changed by modifying self.playerPosition in board.py. However, I haven't figured out how to map coordinates in map.txt to the real position. So I currently just modify the the values and output the screen to see the effects.

• Tensorboard cannot show results from every process, probably due to race condition?