Accelerating Large Language Model Decoding with Speculative Sampling

Models do not generate tokens. They generate probabilities for each token. Inference parameters select a token using those. You can just select the top token all the time or you can do it probabilistically. How you do that in both the speculative decoding and the main inference changes how likely you get the exact same tokens. And then you can choose to accept only if the token matches exactly, or you can choose to accept if it was reasonably likely to be chosen. Let's say the main model picked the 2nd most likely token and speculative picked the most likely. You can reject that - but you get less speed up. You can accept it, you get more speed up, but you do change the output. You risk the distribution of your outputs not being what you hope. I am simplifying. I know in https://arxiv.org/pdf/2302.01318 they specify a probability that you reject a token.

Orders of magnitude seems a bit ambitious. The implementation from the DeepMind paper achieved a 2-2.5X https://arxiv.org/pdf/2302.01318 and most of the tests I've seen [1][2] have been similar, but there are different variations (Medusa, Ouroboros, etc) that can do better/be combined. Recently Together.ai published SpecExec, a SD variant which did claim to get a 10-18X speedups: https://www.together.ai/blog/specexec [1] https://www.reddit.com/r/LocalLLaMA/comments/17h4rqz/specula... [2] https://arxiv.org/pdf/2402.01528v3

I am betting on a combination of quantization and speculative sampling with a distilled smaller set of models: https://arxiv.org/pdf/2302.01318.pdf

Not really. They have a way of squaring this circle, by changing their inference code. Speculative sampling [1] would still make their first claim a lie – sure, there'd still be the original GPT-4 model, plus a smaller draft worker. But early exit decoding [2] allows you to get almost as good results for much cheaper from exactly the same checkpoint. We know that this line of research for large-scale inference is going strong [3] so it stands to reason that OpenAI, with their wealth of talent focused on GPT-4 throughput&inference [4], large contexts and aggressive pricing policy, would also develop something like that. And of course it's "smarter" that way – in a very deceptive sense of the word. 1. https://arxiv.org/abs/2302.01318 2. https://arxiv.org/abs/2207.07061 3. https://arxiv.org/abs/2307.02628 4. https://openai.com/contributions/gpt-4

You can use it as the assistant model to a large model, it's called speculative sampling. You generate text with the small model and validate with the large one, ensuring no deviation occurs. Speedup of 2.5x https://arxiv.org/abs/2302.01318 (DeepMind)

It looks like a description of Speculative Sampling. There's a recent paper from DeepMind about this in the context of LLM [0], although it's not a completely new idea of course. The potential for speedup according to their paper is closer to 2x than 10x however. 0: https://arxiv.org/abs/2302.01318

Speculative sampling to the rescue - you decode locally with a smaller-LLM, and only check from time to time with a large model, like every few tokens. This guarantees exactly the same quality with a big speedup, as you don't need to predict with the large model each individual token. Accelerating Large Language Model Decoding with Speculative Sampling https://arxiv.org/abs/2302.01318

You can use Speculative Sampling, where a draft model is used to generate short continuations of a sequence, which are scored in parallel by the large model. The draft model can be small, and you only need to call the large model from time to time, so you can stream it from SSD or cheap RAM. Using LRDIMM DDR4 at the price of less than $1000 it is possible to stream GPT-3 five times a second, in 4bit quantisation. Multiply that with the 2-2.5x speedup from Speculative Sampling. >Accelerating Large Language Model Decoding with Speculative Sampling https://arxiv.org/abs/2302.01318