Decision Transformer: Reinforcement Learning via Sequence Modeling

> By treating the outcome reward as a predictable label, we reformulate the RLVR problem into a supervised learning task over a score function parameterized by the policy model and optimized using cross-entropy loss. Isn't this how the Decision Transformer works? I don't see it in the references, so I'll be curious to compare the papers in more depth. https://arxiv.org/abs/2106.01345 > By conditioning an autoregressive model on the desired return (reward), past states, and actions, our Decision Transformer model can generate future actions that achieve the desired return. Lately it has crossed my mind that I haven't seen DT brought up much lately, it seemed really interesting when it was first published but I haven't read much follow-up work.

No mention of Decision Transformers or Trajectory Transformers? Both are offline approaches that tend to do very well at long-horizon tasks, as they bypass the credit assignment problem by virtue of having an attention mechanism. Most RL researchers consider these approaches not to be "real RL", as they can't assign credit outside the context window, and therefore can't learn infinite-horizon tasks. With 1m+ context windows, perhaps this is less of an issue in practice? Curious to hear thoughts. DT: https://arxiv.org/abs/2106.01345 TT: https://arxiv.org/abs/2106.02039

> if you include the desired score in your prompt, the model will now strive to produce an answer that is consistent with that score But you need a model to generate score from answer, and then fine-tune another model to generate answer conditioned on score. The first time the score is at the end and the second time at the beginning. It's how DecisionTransformer works too, it constructs a sequence of (reward, state, action) where reward conditions on the next action. https://arxiv.org/pdf/2106.01345 By the same logic you could generate tags, including style, author, venue and date. Some will be extracted from the source document, the others produced with classifiers. Then you can flip the order and finetune a model that takes the tags before the answer. Then you got a LLM you can condition on author and style.

>they've solved reinforcement learning? Transformers can do Reinforcement Learning yes. https://arxiv.org/abs/2106.01345 https://arxiv.org/abs/2205.14953 >they can handle continuous domains, like robot motion? Yes they can handle it just fine. Excellently in fact. https://www.deepmind.com/blog/scaling-up-learning-across-man... https://tidybot.cs.princeton.edu/ https://general-pattern-machines.github.io/ https://wayve.ai/thinking/lingo-natural-language-autonomous-... I don't know if anyone is saying they're the best at or have "solved" everything but they can damn near do anything.

The new hotness this month just says 'screw it; treat RL as a supervised sequence problem with causal transformers.' https://arxiv.org/abs/2106.01345 'We consider the following shift in paradigm: instead of training a policy through conventional RL algorithms like temporal difference (TD) learning [6], we will train transformer models on collected experience using a sequence modeling objective. This will allow us to bypass the need for bootstrapping for long term credit assignment – thereby avoiding one of the “deadly triad” [6] known to destabilize RL. It also avoids the need for discounting future rewards, as typically done in TD learning, which can induce undesirable short-sighted behaviors. Additionally, we can make use of existing transformer frameworks widely used in language and vision that are easy to scale, utilizing a large body of work studying stable training of transformer models.'

PPO is awesome, but so is GPT-style reward-trajectory prediction! http://arxiv.org/pdf/2106.01345v1. As a RL hobbyist, I'd love to see some sort of hybrid approach. Thoughts?