Speculative decoding

Speculative decoding makes a large AI model write faster without lowering the quality of its text. A small, fast model guesses the next few words, and the large model checks all the guesses at once instead of writing every word itself. In December 2024, Google reported using it for AI Overviews in Search. In published tests, text came out two to three times as fast on the same hardware.

How does it work?

A language model writes one token, a word or part of a word, at a time, and every token needs a full run of the model (autoregressive generation). According to Google's researchers, each run reads all of the model's weights from memory, which for a large model can mean on the order of a terabyte per word, while the chip's arithmetic units have capacity to spare. Some tokens are also much easier than others. In "What is the square root of 7? The square root of 7 is 2.646", a small model can copy the second "7" from the question; "2.646" needs the large model. Speculative decoding uses the spare capacity:

1. A small draft model guesses the next few tokens, one after another.
2. The large target model scores all the guesses in one parallel run.
3. The guesses are accepted from left to right. At the first rejected guess, the large model picks that token itself and the remaining guesses are discarded.
4. The next round starts after the last accepted token.

The acceptance test keeps a guess whenever the large model finds it at least as likely as the draft model did, and otherwise keeps it with a probability equal to the ratio of the two. Leviathan et al. prove that the text then follows exactly the distribution of the large model on its own. If the model always takes the most likely token (greedy decoding), the output is even identical.

Where does it come from?

Yaniv Leviathan, Matan Kalman and Yossi Matias of Google published the method in November 2022. With T5-XXL (11 billion parameters) as the target and T5-small (77 million) as the draft model, they measured 2.6 to 3.4 times faster translation and 2.3 to 3.1 times faster summarization on a single TPU v4. At DeepMind, Chen et al. developed the same idea independently as speculative sampling and reported 2 to 2.5 times faster decoding for Chinchilla (70 billion parameters) in a distributed setup, in February 2023. Hugging Face added it to its Transformers library in May 2023 as assisted generation.

What variants are there?

Later methods avoid the separate draft model, which has to be found or trained and takes memory of its own:

• Medusa (Cai et al., 2024) adds extra decoding heads to the model that predict several next tokens; over 2.2 times faster without loss of quality.
• EAGLE (Li et al., 2024) drafts from the model's internal features instead of tokens: 2.7 to 3.5 times lower latency for LLaMA2-Chat 70B.
• Self-speculative decoding (Zhang et al., 2023) drafts with the model itself by skipping some of its layers: up to 1.99 times faster on LLaMA-2.

Where does it go wrong?

• The gain depends on how often the guesses are accepted. Every rejected guess is wasted computation. A larger draft model guesses better but is slower itself: in Leviathan et al.'s translation test, T5-large (800 million parameters) was accepted more often than T5-small, yet the speed-up fell from 3.4 to 1.7 times.
• It needs spare computing power. The vLLM documentation (September 2026) presents it as a way to cut the time between tokens at medium to low request rates, when inference is limited by memory.
• "Lossless" holds up to the precision of the hardware: according to vLLM and Chen et al., floating-point errors can cause slight variations in the output distribution.

Example

In Leviathan et al.'s translation test, T5-small guessed 7 tokens per round and 75% of its guesses were accepted. Equation 1 of the paper gives the expected number of tokens per run of the large model: (1 − 0.75⁸) / (1 − 0.75) ≈ 3.6. The measured speed-up was 3.4 times, a little lower, because the draft model's own runs also take time.