DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

(obviously not all) are incentivized to do. Of course, it's a bit of a verbal compression to claim simply 'scaled up'. They are recognisable scaled up transformers, but most new models come with a few tricks, but we're at the point where those usually are not an architectural rewrite and added to solve an explicit problem, like hallucination, not for big new capability gains.

https://arxiv.org/pdf/2402.03300 That GRPO on code works? > Similarly, for code competition prompts, a compiler can be utilized to evaluate the model’s responses against a suite of predefined test cases, thereby generating objective feedback on correctness Page 4 of https://arxiv.org/pdf/2501.12948

turned up bad coding practices rather than anything malicious. At this pint it would be better for the whole field that these models exist as well as Kimi, Qwen etc as the downward pressure on cost/capabilities leads to commoditisation and the whole race to build a ecogeopolitical moat goes away.

naturally but brings some “language mixing” along. Interestingly, they found a small supervised fine-tuning (SFT) step with language-consistency rewards slightly improved readability, though it came with trade-offs (DeepSeek blog post). My guess is OpenAI has typically used a smaller summarizer model to sanitize reasoning outputs before display (they mentioned summarization/filtering briefly at Dev Day), but perhaps lately they’ve started relaxing that step, causing more multilingual slips to leak through. It’d be great to get clarity from them directly on whether this is intentional experimentation or just a side-effect. [1] DeepSeek-R1 paper that talks about poor readability and language mixing in R1-zero’s raw reasoning https://arxiv.org/abs/2501.12948 [2] OpenAI “Detecting misbehavior in frontier reasoning models” — explains use of a separate CoT “summarizer or sanitizer” before showing traces to end-users https://openai.com/index/chain-of-thought-monitoring/

backwards), and generating the next token based off of that information, the model is activating circuits related to tokens that are not relevant to the next token only, but to specific tokens a handful of tokens after. So, information related to more than just the next upcoming token (including a reference to just one specific token) is being cached during a newline token. Wouldn't call that thinking, but I don't think calling it planning is misguided. Caching this sort of information in the hidden state would be an emergent feature, rather than a feature that was knowingly aimed at by following a specific training method, unlike with models that do test time compute. (DeepSeek-R1 paper being an example, with a very direct aim at turbocharging test time compute, aka 'reasoning'. [1]) The way they went at defining the function of a circuit, was by using their circuit tracing method, which is open source so you can try it out for yourself. [2] Here's the method in short: [3] > Our feature visualizations show snippets of samples from public datasets that most strongly activate the feature, as well as examples that activate the feature to varying degrees interpolating between the maximum activation and zero. > Highlights indicate the strength of the feature’s activation at a given token position. We also show the output tokens that the feature most strongly promotes / inhibits via its direct connections through the unembedding layer (note that this information is typically more meaningful for features in later model layers). [0]: https://transformer-circuits.pub/2025/attribution-graphs/bio... [1]: https://arxiv.org/pdf/2501.12948 [2]: https://github.com/safety-research/circuit-tracer [3]: https://transformer-circuits.pub/2025/attribution-graphs/met...

the model learned to backtrack and correct itself, for example. Makes one wonder what neat math we can find in models - eg. In May 2024, researchers at Anthropic found the Golden Gate Bridge and created a modified Claude that couldn’t stop talking about the bridge: https://www.anthropic.com/news/golden-gate-claude Kind of like the bitter lesson of AI in reverse - instead of us trying to outsmart AI with hand-rolled tricks, why not let it tell us what tricks are out there to be discovered :-)

volumes of chain-of-thought training data; it's left ambiguous exactly how the bulk of this data was generated, though, and it could have been a long process of manual curation. I'd also take the assertion that they started R1 with V3-Base in its release form with a grain of salt. We'll see whether people can reproduce similar results. Either way, there's a new chain-of-thought model with open weights, and that's an incredible achievement regardless of the context - and one that will be meaningful to future research.

to improve efficiency. (last bullet point of page 16, which is the last page of the paper before the bibliography - hmm…) [1] It does even worse on my real world coding problems than the benchmarks would suggest. Some of my tests: write a Qt QAbstractListModel in C++ that parses markdown into block using a C/C++ MD parsing library, write Rust cxx-qt bindings for QTextDocument (all docs included in context), write a window switching script for Wayland with an alternative to wmctrl, etc. I also asked it some geochemistry questions that I had trialed O1 with previously, and the reasoning had a lot hallucinations. The answers were suboptimal to say the least. Having access to the thought process in <think></think> tags is cool but it just reveals how underdeveloped the actual reasoning is compared to whatever o1 is doing. I’ve had it get stuck on silly things like whether a C++ library for markdown exists because I underspecified that I’m okay with header only C libs. O1 has faired much better on my qualitative test suite - even more so when using it via API with a “high” reasoning_effort. With all of the hype over the last few days, it feels like I’m taking crazy pills (loving all the action though). [1] https://arxiv.org/abs/2501.12948