Difference between training and inference

Training is the phase in which an AI model learns: it adjusts its internal settings on a huge set of examples until it makes fewer mistakes. Inference is every use of the finished model after that, such as a chatbot answering a customer. The model's maker pays for training up front; you pay for inference each time you use it, and your questions do not change the model.

What happens in each phase?

Researchers at Google and UC Berkeley describe the two phases like this: training builds an accurate model through an intensive process of repeatedly updating its parameters, and inference uses the trained model to turn inputs into outputs (Patterson et al., 2022). In training, each step runs examples through the model, measures the error with a loss function, works out with backpropagation how much each weight contributed, and adjusts the weights with gradient descent. Inference only runs the input through the model, and the weights stay fixed.

A model can still pick up a task from examples in the prompt. The GPT-3 paper showed this "without any gradient updates or fine-tuning" (Brown et al., 2020), but what the model picks up lasts only for that conversation. Changing the model itself takes another round of training, such as fine-tuning.

Why does training need so much more hardware?

Training has to keep far more in memory than the weights. A GPT-2 model with 1.5 billion parameters needs 3 GB for its weights in 16-bit precision, but training it with the Adam optimizer in mixed precision takes at least 24 GB for the weights, gradients and optimizer states (Rajbhandari et al., Microsoft, 2019). At the largest scale, Meta trained Llama 3 405B on up to 16K H100 GPUs, while one copy of the finished model runs on 16 GPUs across two machines at 16-bit precision (Llama 3 paper). Inference is made cheaper with techniques such as quantization, a KV cache and speculative decoding.

What does each phase cost?

Training is a large cost up front. The Stanford AI Index 2024 estimated the compute for training GPT-4 at $78 million and for Google's Gemini Ultra at $191 million. Inference costs a little per use, but it never stops: by July 2025, ChatGPT users sent 18 billion messages a week (Chatterji et al., 2025). At Google, about three-fifths of the energy used for machine learning from 2019 to 2021 went to inference and two-fifths to training, which Patterson et al. attribute to Google's many services with a billion users each.

For a business that uses a model through an API, the whole bill is inference, paid per token. In September 2026, Anthropic charges $2 per million input tokens and $10 per million output tokens for Claude Sonnet 5 (pricing).

Does using a model train it?

Not while it answers: inference leaves the weights unchanged. Whether a provider later uses your conversations to train new models is set in its data policy. OpenAI, for example, states that it does not use business data from its API and business products for training by default, unless the customer opts in (OpenAI, updated January 2026).

Example

Suppose an accounting firm wants a model to sort incoming invoices into cost categories. Using a ready-made model through an API is inference only: the firm pays per invoice, and the model knows the categories because they are listed in the prompt. If the model keeps putting the firm's own categories in the wrong place, the firm can fine-tune it on invoices it has already sorted. That is a small training run, and after it, every invoice is inference again, on the adjusted model.