Deep learning

Deep learning is a way to build AI in which a computer learns a task from many examples, using a neural network with many layers of simple calculations. It suits tasks that people do without thinking but cannot write down as rules, such as reading handwriting or spotting a product in a photo. Speech recognition, photo search and chatbots such as ChatGPT use it.

Where does the name come from?

The textbook Deep Learning by Goodfellow, Bengio and Courville (2016) describes computers that learn "a hierarchy of concepts", each built from simpler ones. Drawn as a graph, that hierarchy is deep, with many layers, hence the name. In their review in Nature (2015), LeCun, Bengio and Hinton define deep learning as models "composed of multiple processing layers" that learn representations of data "with multiple levels of abstraction". It is a branch of machine learning.

How does it work?

The input layer receives raw data, such as the pixels of an image. Each hidden layer turns the output of the previous one into a slightly more abstract representation, and the output layer gives the answer. LeCun et al. describe what an image model typically learns: edges in the first layer, arrangements of edges in the second, parts of familiar objects in the third, and whole objects after that. The key point, they write, is that "these layers of features are not designed by human engineers". Before deep learning, a system like this needed careful feature engineering by experts.

Training works by trial and correction. The network makes a prediction, a loss function measures the error, backpropagation works out for each weight how the error changes when that weight changes a little, and gradient descent adjusts the weights to reduce the error. A typical system in 2015 could have hundreds of millions of such weights, according to the same review.

Which types are common?

• Convolutional neural networks (CNNs) for images. At the ImageNet competition of 2012, the CNN of Krizhevsky, Sutskever and Hinton had a top-5 error rate of 15.3%, against 26.2% for the runner-up: the correct label was missing from its five best guesses far less often. Three years later LeCun et al. called CNNs "the dominant approach for almost all recognition and detection tasks".
• Recurrent neural networks (RNNs) for sequences such as speech and text. They process a sequence one element at a time.
• Transformers, introduced by Vaswani et al. in 2017, drop that step-by-step processing and rely on attention, which makes training easier to parallelise. They are the basis of large language models such as GPT.

Where does it go wrong?

• It needs a lot of data and computing power. The 2012 ImageNet network learned from 1.2 million labelled images and trained for five to six days on two graphics cards (GPUs). One way around this is to start from a network trained on a large collection and train it further on your own data (fine-tuning). Yosinski et al. (2014) found that the first layers of image networks learn general features that carry over to other datasets and tasks.
• It is hard to explain. Szegedy et al. (2013) showed that a barely visible change to an image can make a network misclassify it, and that the same change can fool a different network trained on other data.

Example

Suppose a webshop wants to sort 20,000 product photos by category. Nobody can write rules for what a sneaker looks like, but a deep learning model can learn it from photos that staff have already sorted. The shop does not have to train a network from scratch: it can take an image model trained on a large photo collection and train it further on its own labelled photos. Someone still checks a sample of the results before the categories go live.