Doing Counterfactual Experience Replay about marllib HOT 9 OPEN

You can try overriding the postprocess_fn in PPOTOrchPolicy. In the MARLlib, such an example may be:

the signature for postprocess_fn is fixed, which is

postprocess_fn(
policy: Policy, sample_batch: SampleBatch, 
other_agent_batches: Optional[Dict[AgentID, SampleBatch]] = None, 
episode: Optional[MultiAgentEpisode] = None
) -> SampleBatch

As displayed in the build_policy_class in https://github.com/ray-project/ray/blob/55fc0710d8472a9abaf244ed6567eb3b13136531/rllib/policy/policy_template.py#L40-L43

You can manipulate the SampleBatch instance provided in the signature to add custom trajectories.

from marllib.

@Aequatio-Space Is this function called before or after both the policy gradient and value function gradient update in ippo or in between?

from marllib.

@nikhil-pitta it is called before both the policy gradient and value function gradient. The pipeline is basically: extra_action_out_fn → postprocess_fn → loss_fn → compute_gradients → apply_gradients.

from marllib.

@Aequatio-Space Thanks so much for the info! We would like to add counterfactual experience to DDPG before adding to the replay buffer or when pulling from the replay buffer to evaluate. Specifically, we want to augment our current step/collected experiences and add to the replay buffer for the algorithm to use. What callback function could we do to use this, and would it be the same for every offline policy? If not a callback, what would you suggest we do?

from marllib.

@nikhil-pitta You mentioned "augment our current step/collected experiences and add to the replay buffer", and it sounds exactly what postprocess_fn do, as our earlier discussion. This extra function applies to all MARLlib algorithms since the interface is identical.

from marllib.

@Aequatio-Space Thank you for the response, and yes that function is exactly what I need! I tried using it for PPO by overriding the postprocess_fn for the IPPOTorchPolicy (https://github.com/Replicable-MARL/MARLlib/blob/master/marllib/marl/algos/core/IL/ppo.py; line 31).

I wanted to do the same thing for IQL, and saw this file https://github.com/Replicable-MARL/MARLlib/blob/master/marllib/marl/algos/core/VD/iql_vdn_qmix.py, but I was unsure how to add a postprocess_fn or any other callback because it looks like a new policy is written up within this file that doesn't accept callback functions like the PPOTorchPolicy.

Do you know what I can do to add callback functionality for IQL?

from marllib.

@nikhil-pitta Note that JointQPolicy inherits the Policy class, which has a function postprocess_trajectory.
https://github.com/ray-project/ray/blob/55fc0710d8472a9abaf244ed6567eb3b13136531/rllib/policy/policy.py#L361-L366.
Directly overriding this function may help.

from marllib.

@Aequatio-Space Thanks for the response again! So now I am trying use IQL, but an issue I'm facing is that I want each of my agents to have separate policies, but I run into the exception ValueError("joint Q learning does not support individual function"), which comes from setting share_policy="individual."

This is not my desired behavior because I would like each agent to have their own policies, which is how I expected IQL to be implemented. Is there any way to get around this?

from marllib.

I remember joint Q learning supports share_policy=all, you can see the related logic at here.

from marllib.