Neural network

A neural network is a computer model that learns from examples instead of following fixed rules. It consists of many small calculating units in layers, linked by adjustable weights. Show it thousands of photos of damaged and undamaged parcels, and it adjusts those weights until it can tell the two apart. The same technique is behind speech recognition and chatbots such as ChatGPT.

How does it work?

Each unit, often called a neuron, takes in numbers, multiplies each by a weight, adds them up with a bias, and passes the result through an activation function, a simple non-linear step. For modern networks the textbook by Goodfellow et al. gives the rectified linear unit (ReLU), which turns negative values into zero, as the default recommendation. The units sit in layers: an input layer takes the data, one or more hidden layers process it, and an output layer gives the result. The middle layers are called hidden because the training data does not say what they should do.

Training works by comparison. The network makes a prediction, a loss function measures how far off it is, and backpropagation works out how much each weight contributed to the error. Gradient descent then moves every weight a small step in the direction that reduces the error, and this repeats over many examples. In their 2015 review in Nature, LeCun, Bengio and Hinton describe deep learning as models "composed of multiple processing layers" whose internal parameters are set with backpropagation.

What does "deep" mean?

It refers to the number of layers. Goodfellow et al. explain the name: "the overall length of the chain gives the depth of the model", and the term deep learning "arose from this terminology". A large language model is a deep network as well. Meta's Llama 3 405B, described in July 2024, has 405 billion parameters in 126 layers and was pre-trained on 15.6 trillion tokens of text (Llama 3 paper). It uses the transformer architecture, one particular way of arranging such layers.

Is it modelled on the brain?

Loosely. Goodfellow et al. call these networks "loosely inspired by neuroscience", and write that neuroscience "is no longer the predominant guide for the field".

Where does it go wrong?

A network learns whatever pattern in the training data predicts the answer, including an unintended one. Geirhos et al. (2020) call this shortcut learning. Their examples include a network that seemed to recognise cows but failed on cows in unusual places such as a beach, because it had learned "grass", and a pneumonia detector for X-rays that had learned to recognise the hospital from a metal token on the scan and did worse on scans from new hospitals. Their recommendation is to test networks on data that differs systematically from the training data.

Example

Suppose a webshop wants to flag fraudulent orders. A neural network gets a set of features per order, such as the amount, the delivery address, the time of day and the payment method, plus thousands of past orders labelled "fraud" or "fine". The hidden layers learn which combinations of these features point to fraud. Accuracy alone says little here: if 1 in 1,000 orders is fraudulent, a model that never flags anything is 99.9% accurate. So you also measure how many fraud cases it catches and how many honest customers it blocks.