What is Backpropagation?
Backpropagation (short for "backward propagation of errors") is the algorithm used to train neural networks. It calculates how much each weight in the network contributed to the prediction error and adjusts those weights to reduce the error.
Why It Matters
Backpropagation is what makes neural network training possible. Without it, there would be no deep learning, no LLMs, no modern AI. Every transformer, CNN, and generative model trained via gradient descent relies on backpropagation to compute the gradients that guide learning.
How It Works
Backpropagation works in tandem with gradient descent through a cycle:
- Forward pass — input data flows through the network layer by layer, producing a prediction.
- Loss calculation — compare the prediction to the actual target using a loss function (e.g., cross-entropy, MSE).
- Backward pass — starting from the output, compute the gradient (partial derivative) of the loss with respect to each weight, using the chain rule of calculus. This propagates error signals backward through the network.
- Weight update — adjust each weight in the direction that reduces the loss, scaled by the learning rate.
- Repeat — iterate over the training data for many epochs until the model converges.
The mathematical foundation is the chain rule: since a neural network is a composition of functions (layer after layer), the derivative of the overall error with respect to any weight can be computed by chaining local derivatives.