Refactor controller to be experiment agnostic about multiagent-rl HOT 3 CLOSED

The core of the controller is how it behaves when receiving each kind of message. Currently, we have the following landscape:

• REQUEST_INIT_MSG: Bookkeeping with agent interval data. Sent when the experiment is starting.
• REQUEST_REGISTER_MSG: More bookkeeping with agent interval data. Sent when the experiment is starting.
• REQUEST_GAME_START_MSG: Bookkeeping with agent game state. Sent when the game is starting.
• POLICY_MSG: Load the policy sent in the message. Sent when the experiment is starting.
• STATE_MSG: Sent by the adapter so the controller can choose an action. Sent while the game is running.
• ACTION_MSG: Sent by the controller with the selected action. Sent while the game is running.
• REQUEST_BEHAVIOR_COUNT_MSG: Get current behavior selection statistics. Sent when the game has finished.
• REQUEST_POLICY_MSG: Never used.

Here is what I think would make sense:

• START_EXPERIMENT_MSG: Bookkeeping to start a new experiment, merging REQUEST_INIT_MSG and REQUEST_REGISTER_MSG. It should work to reset a running experiment as well.
• START_GAME_MSG: Bookkeeping to start a new game, as done by REQUEST_GAME_START_MSG and POLICY_MSG. It should work to reset a running game as well.
• STATE_MSG: Current game state sent while the game is running.
• ACTION_MSG: Selected action sent while the game is running.
• REWARD_MSG: Received reward sent while the game is running.
• FINISH_GAME_MSG: Bookkeeping to finish the current game. This message should also sent the current agent policy, as done by REQUEST_POLICY_MSG.
• FINISH_EXPERIMENT_MSG: Bookkeeping to finish the current experiment. It should stop the controller process.
• ACK_MSG: Sent by the controller after receiving a message that doesn't need to be replied.

I'm not really sure what to do with REQUEST_BEHAVIOR_COUNT_MSG. This kind of message is specific to agents implementing Behavior Q-Learning and should be implemented only by them. Maybe, a message type to log statistics can be generic enough for any agent.

from multiagent-rl.

Commit becbeae is the first effort into refactoring the controller by implementing the messages explained above. Things are still missing though:

• Remove remaining message types from controller - REQUEST_BEHAVIOR_COUNT_MSG, REQUEST_POLICY_MSG and POLICY_MSG - and move their functionality to the agents.
• Game state should be encapsulated in the adapter instead of the controller, so the later can become more decoupled from the Pac-Man simulator.

from multiagent-rl.

There are still two controller behaviors with logic specific to the Pac-Man game:

• _receive_state
• _choose_action

After that, the controller will be pretty much experiment agnostic.

from multiagent-rl.