Overfitting and underfitting

Overfitting and underfitting are the two ways an AI model can learn the wrong lesson from its examples. An overfitted model has learned its examples by heart, including the accidents in them, and does badly on new cases. An underfitted model is too simple to pick up even the main pattern. A sales forecast that copies every bump in last year's figures can still miss next month.

How do you recognise them?

Google's Machine Learning Crash Course compares an overfitted model to an invention that performs well in the lab and is worthless in the real world; an underfitted model does not even do well in the lab. You see the difference by keeping data apart. Track the error, or loss, on the training data and on separate validation data while the model trains. If the two curves first move together and then split, with the validation loss rising, the model is overfitting. If the error stays high on the training data as well, it is underfitting.

What causes it?

Google names two causes of overfitting: training data that does not represent real-world data, and a model that is too complex. The scikit-learn documentation describes the balance with two terms. Bias is a model's average error over different training sets; a model that is too simple has high bias and underfits. Variance is how much the model changes when the training set changes; a model that is too flexible has high variance and overfits. The aim is to keep both low.

How do you prevent it?

Against overfitting:

• More data. It lowers the variance, although scikit-learn advises collecting more only when a simpler model cannot capture the pattern.
• Regularisation, which keeps the model simpler. L2 regularisation in Google's course adds a penalty for large weights; dropout (Srivastava et al., 2014) switches off random units of a neural network during training.
• Early stopping: end training when the validation loss starts to rise.
• Data augmentation: Krizhevsky et al. (2012) enlarged their image set with shifted and mirrored copies, which they call "the easiest and most common method to reduce overfitting on image data".
• A simpler model. According to Google, complex models usually do better on the training set and simple ones on the test set.

Against underfitting, the steps go the other way: a more flexible model, for instance with extra polynomial terms as in the example below, or less regularisation. Google notes that a regularisation rate set too high relative to the learning rate leaves weak weights and poor predictions.

Does this still hold for large models?

Only in part. Zhang et al. (2016) showed that image networks can fit completely random labels, so the classic view does not explain why they still generalise on real data. Belkin et al. (2018) described "double descent": once a model fits its training data exactly, test error can fall again as the model grows. Nakkiran et al. (2019) found the pattern in deep learning, both with model size and with training time. Large language models add a form of their own, memorised test questions. On GSM1k, new maths problems written in the style of an existing benchmark, accuracy fell by up to 8%, and several model families showed signs of systematic overfitting.

Example

In a scikit-learn example, 30 noisy points from a cosine curve are fitted with polynomials of degree 1, 4 and 15. The straight line (degree 1) is too simple for the training points: underfitting. Degree 4 comes close to the true curve. Degree 15 fits the training points almost perfectly and misses the true curve, because it has learned the noise. The cross-validated error in the documentation for version 1.9.1 shows it: 0.41 for degree 1, 0.04 for degree 4 and about 180 million for degree 15.