Scaling laws for LLMs

Scaling laws are formulas, found by experiment, that predict how much better an AI language model gets as it grows: more parameters, more training text and more computing power. They let AI labs estimate a large model's quality from small test runs before paying for the full training. They also explain why makers now train smaller models on far more data, since a smaller model is cheaper to run.

How does it work?

Kaplan et al. (OpenAI, 2020) trained transformer language models of many sizes and measured their loss: the cross-entropy on the next token, a score for how badly the model predicts text (see loss function). The loss fell as a power law of three factors: the number of parameters, the number of training tokens and the training compute, as long as the other two were not the bottleneck. Some trends held over more than seven orders of magnitude, while the shape of the model, such as its depth or width, mattered little.

A power law means that every tenfold increase cuts the loss by the same percentage. For parameters, Kaplan et al. fitted L(N) = (Nc / N) ^ 0.076, so ten times more parameters gives about 16% less loss; for data the exponent was 0.095, about 20% per tenfold. They saw no sign of the trends levelling off, but noted that they must level off eventually, because natural language is never perfectly predictable.

What did Chinchilla change?

Kaplan et al. concluded that extra compute should go mainly into a bigger model. Hoffmann et al. (DeepMind, 2022) trained more than 400 models and found that model size and training data should grow at the same rate: double the model, double the tokens. Most large models of the time, including GPT-3 and Gopher, had been trained on about 300 billion tokens, too few for their size. DeepMind tested this with Chinchilla: the compute budget of the 280-billion-parameter Gopher, spent on a model with 70 billion parameters and 1.4 trillion tokens. Chinchilla outperformed Gopher and GPT-3 on a large range of tasks and, being smaller, was cheaper to use. The paper's projections work out to about 20 tokens per parameter. The authors trace part of the gap with Kaplan et al. to one choice: the OpenAI team used the same number of training tokens and the same learning-rate schedule for every model.

What can they predict?

For GPT-4, OpenAI predicted the final loss accurately from models trained with at most 10,000 times less compute, and the pass rate on part of the HumanEval coding test from models with at most 1,000 times less (GPT-4 technical report, March 2023). The same report says that certain capabilities remain hard to predict. Meta's Llama 3 paper (July 2024) adds that scaling laws usually predict the loss, not scores on specific tests. Wei et al. (2022) call an ability emergent when larger models have it and smaller ones do not, so extrapolation cannot predict it; Schaeffer et al. (2023) argue that many of those jumps come from the metric researchers chose.

Example

At Chinchilla's ratio, a model with 8 billion parameters would be trained on about 160 billion tokens. Meta pre-trained Llama 3 8B on more than 15 trillion (model card, April 2024), more than 1,800 tokens per parameter. The Llama 3 paper gives the reason: Meta trains its smaller models far longer than is compute-optimal, and they perform better than compute-optimal models at the same inference budget. Sardana et al. (2023) reach the same conclusion for any model that expects many requests, around a billion, and saw quality keep improving up to 10,000 tokens per parameter.