Loss function

A loss function is the score that tells an AI model during training how wrong its answers are. The model makes a prediction, the loss function compares it with the right answer and turns the difference into one number, and training adjusts the model to lower that number. What you count as a mistake shapes what the model learns, such as whether one big error outweighs several small ones.

How does it work?

The textbook Deep Learning by Goodfellow, Bengio and Courville treats objective function, criterion, cost function, loss function and error function as names for the same thing: the function that training minimises. Google's Machine Learning Crash Course describes loss as a number for how wrong a model's predictions are, the distance between the predictions and the actual labels. During training the loss is computed over a batch of examples, backpropagation works out how each weight affects it, and gradient descent moves the weights to lower it.

The loss is often a stand-in for what you really care about. Chapter 8 of the textbook calls this a surrogate loss: the number of wrongly classified examples cannot be minimised efficiently with gradients, so a classifier is usually trained on the negative log-likelihood of the correct class instead.

Which loss functions are common?

• Mean squared error (MSE) and mean absolute error (MAE), for predicting numbers such as a price or a delivery time. MSE squares each error, so it penalises outliers more heavily than MAE. Chapter 6 of the textbook shows what that means: with enough data, a model trained on MSE predicts the mean for each input, and one trained on MAE the median.
• Cross-entropy, for choosing between classes, such as spam or not spam, or the next token in a text. According to the textbook, most modern neural networks are trained with maximum likelihood, which amounts to minimising the cross-entropy between the training data and the model's predictions. Kaplan et al. (OpenAI, 2020) trained language models on this loss, averaged over a context of 1,024 tokens, and used it as their main measure of performance (scaling laws).
• Contrastive loss, for embeddings: it pulls the vectors of matching pairs together and pushes the others apart.

PyTorch ships these as ready-made classes, among them nn.MSELoss, nn.L1Loss (MAE), nn.CrossEntropyLoss and nn.BCEWithLogitsLoss, which combines a sigmoid with binary cross-entropy for yes/no labels.

Where does it go wrong?

• The training loss keeps falling while the model gets worse on new data: overfitting. Google's crash course shows it as two loss curves that diverge: after a while the loss on the training set levels off or keeps dropping, and the loss on the validation set rises.
• The loss goes up instead of down. When the learning rate is too large, gradient descent can increase the training error (Deep Learning, chapter 11).
• The gradients become too small. The textbook notes that MSE and MAE often give poor results with gradient-based training when output units saturate, one reason cross-entropy is more popular.

Example

Suppose a model estimates the selling price of three houses and is off by €10,000, €10,000 and €40,000. The mean absolute error is €20,000. The mean squared error is (10,000² + 10,000² + 40,000²) / 3 = 600,000,000 squared euros; its square root, the RMSE, is about €24,500. The one large miss weighs more in the MSE, so a model trained on it works harder to avoid large misses. For a language model the idea is the same with probabilities: if it gives the correct next token a probability of 10%, the cross-entropy loss for that token is −ln 0.1 ≈ 2.3; at 90% it is about 0.1.