Comments (6)
Hi, Jason
As a combination of the transformer and Q network, I would like to point out that the memory module is critical in the implementation of the combination. The practical experience shows that a naive attention mechanism should incorporate memory, such as historical observation or reward, to work well.
from dtqn.
Hi Jason,
I'm sorry, I didn't see your message. Thanks for your interest in DTQN! I agree, I think transformers for reinforcement learning is an understudied field. One of the reasons for this could be that transformers are most well-suited to solve POMDPs, which is not as popular as fully-observable MDPs. Probably the 2 biggest papers for transformers in RL are Decision Transformer, which is for offline RL solving MDPs, and GTrXL. Neither of which use a DQN-like approach. There is also Stabilizing Q-learning with ConvNets and ViT, which uses the vision transformer as an image encoder, and then DQN for action selection; this is a different use case for the transformer compared to DTQN and the 2 previous works.
In general, transformers in RL is becoming more and more popular. In particular, roboticists are leveraging the ability to use multi-modal data in transformers (thanks PerceiverIO!), such as in PerAct. The Perceiver archiecture allows us to easily attend across modalities (for instance, an image of the scene and a language goal). That is a very powerful ability, which I think will become more popular. For sure, this is an exciting time to be interested in transformers.
from dtqn.
Hi, Jason As a combination of the transformer and Q network, I would like to point out that the memory module is critical in the implementation of the combination. The practical experience shows that a naive attention mechanism should incorporate memory, such as historical observation or reward, to work well.
Thanks for your insight. Do you ever seen some work that combination of the transformer and offline RL algorithm?
from dtqn.
Thanks for your replay. I thought Decision Transformer is more like Sequential-Modeling , GTrXL is more like MDPs-solver. And DTQN is also like MDPs-solver. Have you ever seen some work about offline RL algorithm with Transformer?
from dtqn.
Decision Transformer was built to solve MDPs using offline RL. Really, it's trained in a supervised way using a dataset of transitions from the D4RL repo. GTrXL and DTQN both incorporate memory by using a transformer where each token is a past observation, so they can solve POMDPs. While GTrXL primarily focused on solving MDPs, it can also be applied for POMDPs. In contrast, DTQN is built for solving POMDPs.
from dtqn.
Decision Transformer was built to solve MDPs using offline RL. Really, it's trained in a supervised way using a dataset of transitions from the D4RL repo. GTrXL and DTQN both incorporate memory by using a transformer where each token is a past observation, so they can solve POMDPs. While GTrXL primarily focused on solving MDPs, it can also be applied for POMDPs. In contrast, DTQN is built for solving POMDPs.
Thanks for your insight. It's very helpful for me.
from dtqn.
Related Issues (11)
- Reproduction of GridVerse results HOT 9
- Getting an error stating environment D does not exist HOT 1
- Error run.py HOT 2
- I'm trying to connect DTQN network with SUMO. I want to know inputs of DTQN. HOT 2
- `in-embed` instead of `inembed`? HOT 3
- Can this project run on the Windows? HOT 1
- Replacing `done` with `truncated` and `terminated` HOT 6
- Rendering issue - ImportError HOT 1
- Matrix size mismatch error HOT 7
- paper reproduction in gridverse 7x7, 9x9 HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dtqn.