Comments (9)
You can try overriding the postprocess_fn
in PPOTOrchPolicy. In the MARLlib, such an example may be:
MARLlib/marllib/marl/algos/core/CC/coma.py
Lines 116 to 125 in 368c617
the signature for postprocess_fn
is fixed, which is
postprocess_fn(
policy: Policy, sample_batch: SampleBatch,
other_agent_batches: Optional[Dict[AgentID, SampleBatch]] = None,
episode: Optional[MultiAgentEpisode] = None
) -> SampleBatch
As displayed in the build_policy_class
in https://github.com/ray-project/ray/blob/55fc0710d8472a9abaf244ed6567eb3b13136531/rllib/policy/policy_template.py#L40-L43
You can manipulate the SampleBatch
instance provided in the signature to add custom trajectories.
from marllib.
@Aequatio-Space Is this function called before or after both the policy gradient and value function gradient update in ippo or in between?
from marllib.
@nikhil-pitta it is called before both the policy gradient and value function gradient. The pipeline is basically: extra_action_out_fn → postprocess_fn → loss_fn → compute_gradients → apply_gradients.
from marllib.
@Aequatio-Space Thanks so much for the info! We would like to add counterfactual experience to DDPG before adding to the replay buffer or when pulling from the replay buffer to evaluate. Specifically, we want to augment our current step/collected experiences and add to the replay buffer for the algorithm to use. What callback function could we do to use this, and would it be the same for every offline policy? If not a callback, what would you suggest we do?
from marllib.
@nikhil-pitta You mentioned "augment our current step/collected experiences and add to the replay buffer", and it sounds exactly what postprocess_fn
do, as our earlier discussion. This extra function applies to all MARLlib algorithms since the interface is identical.
from marllib.
@Aequatio-Space Thank you for the response, and yes that function is exactly what I need! I tried using it for PPO by overriding the postprocess_fn for the IPPOTorchPolicy (https://github.com/Replicable-MARL/MARLlib/blob/master/marllib/marl/algos/core/IL/ppo.py; line 31).
I wanted to do the same thing for IQL, and saw this file https://github.com/Replicable-MARL/MARLlib/blob/master/marllib/marl/algos/core/VD/iql_vdn_qmix.py, but I was unsure how to add a postprocess_fn or any other callback because it looks like a new policy is written up within this file that doesn't accept callback functions like the PPOTorchPolicy.
Do you know what I can do to add callback functionality for IQL?
from marllib.
@nikhil-pitta Note that JointQPolicy
inherits the Policy
class, which has a function postprocess_trajectory
.
https://github.com/ray-project/ray/blob/55fc0710d8472a9abaf244ed6567eb3b13136531/rllib/policy/policy.py#L361-L366.
Directly overriding this function may help.
from marllib.
@Aequatio-Space Thanks for the response again! So now I am trying use IQL, but an issue I'm facing is that I want each of my agents to have separate policies, but I run into the exception ValueError("joint Q learning does not support individual function"), which comes from setting share_policy="individual."
This is not my desired behavior because I would like each agent to have their own policies, which is how I expected IQL to be implemented. Is there any way to get around this?
from marllib.
I remember joint Q learning supports share_policy=all
, you can see the related logic at here.
MARLlib/marllib/marl/algos/run_vd.py
Lines 105 to 118 in 368c617
Try to adapt the code under this setting.
from marllib.
Related Issues (20)
- How ma_policy works? and How I can use it?
- "MulBackward0 returned nan values" error when launch HATRPO after HAPPO
- How MARLlib works with ray algorithms like PPO? HOT 1
- Update to newest RLlib version? HOT 5
- JointQ not work in custom env HOT 1
- Trouble implementing custom environment HOT 3
- Episodes_this_iter parameter
- Supporting Individual Action Spaces
- How to export trained model as a .pt (pytorch ) or ONNX model. HOT 2
- How do I get an agent's position in the environment in the `postprocess_trajectory` method? HOT 1
- IQL setup for Custom Env
- Can the qmix algorithm solve the AirCombat problem and does Marllib support it?
- Training agents with IQL HOT 3
- Query Regarding num_workers Setting Resulting in Multiple Concurrent Environments During Training
- Inferencing the learned Policies HOT 2
- Inter-agent communication before compute_actions HOT 1
- How to set up exploration strategies for Agents?
- Confusing results in simple spread environment HOT 1
- Zero reward in Overcooked environment regardless of algorithm/length of training
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from marllib.