When Pacman is finishing the game, ghosts may rush to capture it otherwise the penalty would be too great.

It'll be necessary to update the documentation.

The weights vector determines the importance for each feature in the estimated value for the current state. However, by normalizing it, we actually change the feature importance in relation with other actions.

Possibilities:

• Remove normalization at all.
• Normalize all weights instead of only the action weights.

Currently, weights is a dictionary that maps actions (or behaviors) to weights array, one for each feature, i.e. action -> [feature].

Instead, it must map an action to a feature and, then, to the weight, i.e. action -> feature -> weight. Then, should the features order change from a previous game to another one, where the policy is loaded, it'll map correctly the action to its weight.

Currently, ghosts receive the negative of Pacman reward. This means that each game step where the Pacman does nothing, it will receive +1 reward, making the ghosts simply wait the Pacman to get lost and penalize itself.

It would be interesting that ghosts receive the following rewards:

• +500: Capturing the Pacman
• -500: Getting captured
• -1: Otherwise

Currently, each agent receives only its own position from the simulation. Instead, agents must receive the positions from every other agent, since it's necessary for cooperative learning.

Testing with different conditions allows to understand how well policies learned with the same algorithm perform in different scenarios. We must test with the Pacman alone up to 4 ghosts in the same map.

For instance:

• Pacman and ghost agents class
• Layout file
• Date and time (either inside the file and in its name)

So far, only enemy distances are used as a feature. Do the same for allies.

This guarantees learning convergence.

Simulator script must accept a CLI flag, such as -g or --gui to output game in a GUI.

Each game step, we must log the policy weights for future analysis.

For instance, multiple plots in the same figure.

Currently, the simulator does not communicate any food position at all. This is desirable since it influences the game score.

The punishment for losing the game may be too high, making the agent adopt a conservative strategy.
By reducing the punishment or increasing the reward for winning games, the agent may become greedier.

Currently, three files are generated during a simulation: learn_scores.txt, test_scores.txt, and pacman_policy.txt. These files should be unified in a single file, such as the one below.

[learn_scores]
10.1
1.1
-25.3
[test_scores]
0.0
1500.2
[pacman_policy]
0.1,-20.5,30.2
2,70,-100

By visual inspection, Pacman sometimes stops at corners and take a while to escape from it. This behavior may be due to a wrong implementation of flee_behavior or a sequence of Stop actions.

Each game step, we must log the selected behavior for future analysis.

Currently, all features are implemented in agents module, which is simply wrong. Let's create a new module for it.

So far, only the Pacman is learning. Let's test how ghosts can learn.

With this behavior, Pacman would be able to pursue ghosts. Hopefully, it'll only activate when they are white.

After transforming walls into a property, paths are recalculated whenever the walls change. However, walls change at each game step, MAS receiving it from the message sent by the simulator. This severely degrades system performance.

A deterministic version of Q-learning must be implemented, such as the one used in example_predator_prey.py.

Several scripts, modules, and packages were used for testing purposes but are still committed to GitHub, such as example_predator_prey.py, bayes, and references. These files must be removed from the project.

In testing mode, exploration rate drops to 0, however the agent keeps used rewards to learn. We must change it so as during test mode, agents don't learn anymore.

Since an INIT message destroys current agents and build new ones, it also destroys the learned policy. Hence every game a new agent plays without any previous knowledge.

Solution: separate INIT (agent creation) from START(game beginning). Then, INIT happens once in the simulation.py script whereas START happens at the beginning of each game.

One of the main goals is that this code be reused, so a tutorial with lots of examples is required.

1. How does this work again? A simple explanation on how to use the system, and how it works in broad terms (how systems communicate).
2. Wait, what? A simple explanation on how this code is organized and which "conceptual blocks" (adapter system, controller system) are associated to which code files.
3. It's simple, we kill the Pac-Man Simple explanations on how to create and include a new ghost agent (say, one whose only action is to turn right on corners).
4. I know Kung Fu A simple tutorial on how to use a new learning method for the agent (different from Q-learning). It assumes, of course, that the method is already implemented, it just needs to be incorporated.
5. A Brave New World A simple tutorial on how to use a different simulator (like Platformer AI).
6. Hasta la vista, Baby Ideas on how to adapt the system to communicate through ROS.

It is useful to plot a histogram with data collected after running several trials that generated different policies. This statistic may provide whether most policies perform well or underperform.

Currently, agents can't learn proper policies because the state space is quite large. Therefore, it is quite difficult for an agent to reach the same state over and over again (a condition necessary for proper learning).

Q-learning with function approximation best fits this scenario and, probably, may provide learning capabilities for the agents.

I'm having some troubles in creating an agent that can win the game taking the ghosts into account. Thus, let's take a step back and simplify things.

• Remove all ghosts from the layout
• Implement intelligence that allows the Pacman to finish the game by eating all food

Regression allows better interpretation of collected data, with better analysis of the general trend.

Learning phase has both exploration and exploitation. Testing phase only exploit learned results.

Now that Pacman can properly win the game with quasi-optimal behavior, it must learn to win in the presence of another ghost.