AI inference

Inference is the moment a trained AI model is put to work: it takes new input, such as a customer's question, and produces an answer. Training is how the model learned; inference is every use after that. Every chatbot reply is one. For a business, inference is what you pay for with each use, often per token, a small piece of text, and it decides how quickly answers arrive.

How is inference different from training?

Researchers at Google and UC Berkeley describe the two phases like this: training builds an accurate model through an intensive process of repeatedly updating its parameters, and inference uses the trained model to generate outputs from inputs (Patterson et al., 2022). During inference the weights stay fixed. The GPT-3 paper showed that a model can pick up a task from examples in the prompt "without any gradient updates or fine-tuning" (Brown et al., 2020), but what it picks up lasts only for that conversation. To make a model use your own information, you add it to the prompt each time, for example with RAG, or train it further with fine-tuning. The entry training vs inference compares the two in more detail.

Training is a large cost up front. Inference costs a little each time, and that adds up: at Google, about three-fifths of the energy used for machine learning from 2019 to 2021 went to inference and two-fifths to training, according to the same paper.

What happens during inference?

In a large language model, inference has two phases. Pope et al. (Google, 2022) call the processing of the input tokens "prefill" and the token-by-token writing of the answer "decode". Prefill reads the whole prompt in one parallel pass. Decode needs a full pass through the model for every new token (autoregressive generation), and a KV cache keeps intermediate results of earlier tokens so they are not computed again. Two speed measures follow from this. Time to first token is the time from sending the prompt until the first token of the answer arrives (Anthropic). After that, speed is counted in tokens per second.

How is it made faster and cheaper?

• Batching: a server handles many requests at once. The open-source serving library vLLM stores the KV cache in pages, the way an operating system manages memory, and its authors measured 2 to 4 times the throughput of earlier systems at the same latency (Kwon et al., 2023).
• Quantization: the weights are stored with fewer bits, so the model needs less memory.
• Speculative sampling: a small model drafts a few tokens and the large model checks them in one pass. On Chinchilla, a model with 70 billion parameters, DeepMind measured 2 to 2.5 times faster decoding without loss of quality (Chen et al., 2023). Leviathan et al. (Google, 2022) describe the same idea as speculative decoding.
• A smaller model: Anthropic's documentation calls choosing the right model one of the most direct ways to reduce latency.

Example

Suppose a news site uses a language model to summarise, tag and write SEO metadata for 5,000 articles a day, with about 1,800 input tokens and 400 output tokens per article. That is 9 million input tokens and 2 million output tokens a day. In September 2026, Anthropic charges $2 per million input tokens and $10 per million output tokens for Claude Sonnet 5 (pricing), so the day costs $18 plus $20, or $38. Tagging alone, at 400 input and 50 output tokens per article, takes 2 million input and 250,000 output tokens: $6.50 on Sonnet 5, or $3.25 on Claude Haiku 4.5 at $1 and $5 per million. Because the articles do not need an instant answer, the site can also use the Batch API, which halves the price of both input and output tokens.