Question about transformer with DQN about dtqn HOT 6 CLOSED

Hi, Jason
As a combination of the transformer and Q network, I would like to point out that the memory module is critical in the implementation of the combination. The practical experience shows that a naive attention mechanism should incorporate memory, such as historical observation or reward, to work well.

from dtqn.

Hi Jason,

I'm sorry, I didn't see your message. Thanks for your interest in DTQN! I agree, I think transformers for reinforcement learning is an understudied field. One of the reasons for this could be that transformers are most well-suited to solve POMDPs, which is not as popular as fully-observable MDPs. Probably the 2 biggest papers for transformers in RL are Decision Transformer, which is for offline RL solving MDPs, and GTrXL. Neither of which use a DQN-like approach. There is also Stabilizing Q-learning with ConvNets and ViT, which uses the vision transformer as an image encoder, and then DQN for action selection; this is a different use case for the transformer compared to DTQN and the 2 previous works.

In general, transformers in RL is becoming more and more popular. In particular, roboticists are leveraging the ability to use multi-modal data in transformers (thanks PerceiverIO!), such as in PerAct. The Perceiver archiecture allows us to easily attend across modalities (for instance, an image of the scene and a language goal). That is a very powerful ability, which I think will become more popular. For sure, this is an exciting time to be interested in transformers.

from dtqn.

Hi, Jason As a combination of the transformer and Q network, I would like to point out that the memory module is critical in the implementation of the combination. The practical experience shows that a naive attention mechanism should incorporate memory, such as historical observation or reward, to work well.

Thanks for your insight. Do you ever seen some work that combination of the transformer and offline RL algorithm?

from dtqn.

Thanks for your replay. I thought Decision Transformer is more like Sequential-Modeling , GTrXL is more like MDPs-solver. And DTQN is also like MDPs-solver. Have you ever seen some work about offline RL algorithm with Transformer?

from dtqn.

Decision Transformer was built to solve MDPs using offline RL. Really, it's trained in a supervised way using a dataset of transitions from the D4RL repo. GTrXL and DTQN both incorporate memory by using a transformer where each token is a past observation, so they can solve POMDPs. While GTrXL primarily focused on solving MDPs, it can also be applied for POMDPs. In contrast, DTQN is built for solving POMDPs.

from dtqn.

Decision Transformer was built to solve MDPs using offline RL. Really, it's trained in a supervised way using a dataset of transitions from the D4RL repo. GTrXL and DTQN both incorporate memory by using a transformer where each token is a past observation, so they can solve POMDPs. While GTrXL primarily focused on solving MDPs, it can also be applied for POMDPs. In contrast, DTQN is built for solving POMDPs.

Thanks for your insight. It's very helpful for me.

from dtqn.