Autoregressive generation

Autoregressive generation is how chatbots such as ChatGPT write: one small piece of text at a time, each time choosing the next piece based on everything written so far. It works like the word suggestions on your phone keyboard, applied over and over. That is why an answer appears on screen word by word, and why a long answer takes longer and costs more than a short one.

How does it work?

A large language model works with tokens, pieces of words. The transformer paper by Vaswani et al. (2017) describes it in one sentence: "At each step the model is auto-regressive, consuming the previously generated symbols as additional input when generating the next." In practice:

1. The model reads the whole prompt in one pass. All input tokens are known at the start, so they can be processed in parallel; Pope et al. (Google, 2022) call this step prefill.
2. It computes a probability for every token in its vocabulary as the next one.
3. It picks one: always the most likely token (greedy decoding), or a sample shaped by temperature or top-p.
4. The chosen token is appended, and the longer sequence becomes the input for the next step.
5. This repeats until the model produces a stop token or reaches the length limit.

Jay Alammar shows the same loop for GPT-2 and notes the trade-off: BERT is not autoregressive, and in exchange can use the context on both sides of a word. Diffusion language models are another alternative: LLaDA (Nie et al., 2025) predicts masked tokens in parallel and refines them, and performed comparably to autoregressive baselines of its own.

Why is it slow, and how is it sped up?

Every new token needs a full pass through the model. In the words of Leviathan et al. (Google, 2022), "decoding K tokens takes K serial runs of the model". Two techniques reduce the cost:

• The KV cache keeps the attention keys and values of earlier tokens in memory, so each step only has to compute them for the new token.
• Speculative decoding lets a small, fast model draft several tokens that the large model then checks in one parallel pass. Leviathan et al. measured a 2 to 3 times speed-up on T5-XXL with identical output.

Output is also priced higher than input. In September 2026 Anthropic charges $3 per million input tokens and $15 per million output tokens for Claude Sonnet 4.6.

Where does it go wrong?

A token, once written, stays, and it becomes context for everything after it. Bengio et al. (2015) point to the mismatch behind this: in training the model sees the correct previous token, at generation time its own. That "discrepancy between training and inference can yield errors that can accumulate quickly along the generated sequence."

Always picking the most likely token has its own problem. Holtzman et al. (2019) found that it leads to text that is "bland and strangely repetitive", and proposed nucleus sampling, better known as top-p.

Example

Suppose a customer asks a webshop assistant when the shop is closed, and the answer so far reads "We are closed on". The model scores every token in its vocabulary, and "Sunday" may come out on top. That token is appended, the model runs again on "We are closed on Sunday", and picks the next token, perhaps a full stop. Each step is one pass through the model, and with streaming each token reaches the screen the moment it is chosen. Where one token ends and the next begins differs per model.