What is a Loss Function?

What is a Loss Function?

A loss function (also called a cost function or objective function) is a mathematical function that measures how far a model's predictions are from the actual target values. It provides the error signal that training algorithms use to improve the model.

Why It Matters

The loss function defines what "learning" means for a model. It's the quantity that training minimizes. Choosing the right loss function is crucial: it shapes what the model optimizes for and directly affects model behavior. For LLMs, the cross-entropy loss on next-token prediction is what drives the model to learn language.

How It Works

A loss function takes two inputs:

• Prediction — what the model output
• Target — what the correct answer is

It returns a single number (the loss) representing how wrong the prediction is. Training algorithms (gradient descent + backpropagation) then adjust model weights to minimize this number.

Common loss functions:

• Cross-entropy loss — standard for classification and language modeling. Measures the difference between predicted probability distribution and actual distribution.
• Mean Squared Error (MSE) — standard for regression. Averages the squared differences between predictions and targets.
• Binary cross-entropy — for binary classification (yes/no, spam/not spam).
• Contrastive loss — for learning embeddings, pushing similar items together and dissimilar items apart.
• KL divergence — measures how one probability distribution differs from another, used in variational autoencoders and distillation.

Example

When training GPT on next-token prediction, the cross-entropy loss measures: "The model predicted 'cat' with 10% probability but the actual next word was 'cat'." The loss is high because the model was not confident in the right answer. After many training steps, the model learns to assign higher probability to correct tokens, driving the loss down.