Batch size and learning rate

Batch size and learning rate are two settings that control how an AI model learns during training. The batch size is how many examples the model looks at before it adjusts itself. The learning rate is how big each adjustment is. You meet both when you fine-tune a model on your own examples: too high a learning rate can make the model worse, too low makes training slow.

How do they work?

Training improves a model's weights in small steps. For each step, the model takes a batch of examples, backpropagation computes the average gradient over that batch, and gradient descent moves every weight against the gradient: weight = weight - learning_rate * gradient.

The batch size decides how good that gradient estimate is and how much work one step takes. The textbook Deep Learning by Goodfellow, Bengio and Courville notes that a larger batch gives a more accurate estimate, with less than linear returns: 100 times as many examples reduce the error of the estimate only by a factor of 10. Memory usually sets the upper limit, and typical sizes are powers of 2 from 32 to 256, which GPUs often process faster.

The learning rate is, according to the same book, perhaps the most important hyperparameter: if you can tune only one, tune this one. When it is too large, gradient descent can increase the error instead of reducing it. When it is too small, training is slower and can get stuck at a high error.

How do the two interact?

Goyal et al. (Facebook, 2017) proposed the linear scaling rule: when you multiply the batch size by k, multiply the learning rate by k. It held in their ImageNet experiments up to batches of about 8,000 images, beyond which accuracy dropped quickly. Smith et al. (Google Brain, 2017) showed a related effect: instead of lowering the learning rate during training, you can raise the batch size and reach about the same accuracy with fewer weight updates.

The Deep Learning Tuning Playbook by researchers at Google treats the batch size as a question of training speed and cost: with the other settings re-tuned, any batch size should reach the same final result. It advises the largest batch the hardware handles efficiently, and a new learning-rate search whenever the batch size changes. It advises against gradient accumulation, which adds up several small batches before one update, because that gives no speed gain.

Does the learning rate change during training?

Usually it follows a schedule. Warmup ramps it up over the first steps, which the playbook recommends against early instability. After that it decays, and the playbook prefers a straight line or a cosine curve. The textbook explains why it has to come down: the noise of sampling a batch does not vanish near the minimum. GPT-3 was trained with a linear warmup over the first 375 million tokens and cosine decay to 10% of the peak (Brown et al., 2020). Llama 2 used 2,000 warmup steps and the same decay to 10% (Touvron et al., 2023).

For fine-tuning, the BERT authors suggest trying learning rates of 5e-5, 3e-5 and 2e-5 with batch sizes of 16 or 32 (Devlin et al., 2018), and Meta fine-tuned Llama 2 with a learning rate of 2e-5 and batches of 64. Hosted services can choose for you: in September 2026, OpenAI's fine-tuning API takes a batch size and a learning-rate multiplier, both "auto" by default (openai-python).

Example

Goyal et al. trained the image recognition network ResNet-50 on ImageNet. With 256 images per batch and a learning rate of 0.1, the model's first guess was wrong for 23.60% of the images it had not seen in training. With 8,192 images per batch and the learning rate scaled to 3.2, the error was 23.74% with a gradual warmup and 24.84% without one. The large batch let them spread the work over 256 GPUs and train in one hour.