s1: Simple test-time scaling

> My favorite trick for controlling the reasoning level is the hack First described here I think, specifically for Qwen models: https://arxiv.org/abs/2501.19393, although it seems unclear how much it actually improves the final response unless you create a new fine-tuned model with this in mind, and also if it's possible to replicate for other models. Still, got inspired and gonna give it a try with DiffusionGemma although slightly differently

This is the same reasoning behind why Yann Lecun thought test-time scaling would not work for LLMs: compounding error. Instead, the more tokens LLMs use, the better their performance on many tasks. LLMs can self-correct, evidenced by the power of getting models to question themselves by emitting "Wait," in S1. https://arxiv.org/abs/2501.19393

>One theory is that the knowledge required to solve the task is already stored in the parameters of the model, and only the style has to change for task success >In particular, learning to generate longer outputs may be possible in few parameters Reminded me of: https://arxiv.org/abs/2501.19393 >we develop budget forcing to control test-time compute by forcefully terminating the model’s thinking process or lengthening it by appending “Wait” multiple times to the model’s generation when it tries to end. This can lead the model to double-check its answer, often fixing incorrect reasoning steps Maybe, indeed, the model simply learns to insert the EOS token (or similar) later, and the capability is already in the base model

a paper with a similar idea on scaling test time reasoning, this is sorta how all the thinking models work under the hood. https://arxiv.org/abs/2501.19393

I don’t think you have an accurate understanding of how LLMs work. https://arxiv.org/abs/2501.19393 These tokens DO extend the thinking time. We are talking about causal autoregressive language models, and so these tokens can be used to guide the generation.

If that's an issue, there's a workaround using structure generation to force it to output a </thiking> token after some threshold and force it to write the final answer. It's a method used to control thinking token generation showcased in this paper: https://arxiv.org/abs/2501.19393

Yes, the authors explicitly highlighted those two points in the abstract, in terms of them being the elicitation threshold for complex reasoning, namely, an extremely complete pre-trained foundation model, and a set of extremely high quality examples post-training. To your question on finetuning on the initial 10 million pool - intuitively, it would require tremendous amount of finetuning data to move the needle - you really won't be able to move the gradients much with just 817 examples, that initial pool is effectively enforcing pretty rigid regularization. There is now an increasing interest in showing that small data with inference time scaling is providing significant yield. Couple of recent examples: * TinyZero: https://github.com/Jiayi-Pan/TinyZero * s1 Simple Test Time Scaling: https://arxiv.org/abs/2501.19393

nope you have the article right. "if you copy an existing model, you can get a pretty comparable model!" Really, [what the underlying paper](https://arxiv.org/abs/2501.19393) shows is how much you can improve existing models with several of the techniques released by openai recently (extending thought processes via "wait") It shows we can get a lot of mileage out of that technique.

Agreed. Here are three things that I find surreal about the s1 paper. (1) The abstract changed how I thought about this domain (advanced reasoning models). The only other paper that did that for me was the "Memory Resource Management in VMware ESX Server". And that paper got published 23 years ago. (2) The model, data, and code are open source at https://github.com/simplescaling/s1. With this, you can start training your own advanced reasoning models. All you need is a thousand well-curated questions with reasoning steps. (3) More than half the references in the paper are from 2024 and Jan 2025. Just look at the paper's first page. https://arxiv.org/pdf/2501.19393 In which other field do you see this?