Activation function

An activation function is a small rule inside a neural network, the kind of AI model behind chatbots and image recognition. It decides how much of each signal passes on to the next step. Without it the model could only learn patterns that follow a straight line, like "twice the order, twice the shipping cost". You never choose one as a user: whoever builds the model does.

Why does a neural network need one?

A neural network works in layers. Each layer multiplies its inputs by weights and adds them up, which is a linear step. Two linear steps in a row are again one linear step: w₂(w₁x + b₁) + b₂ is simply (w₂w₁)x + (w₂b₁ + b₂). The Deep Learning textbook by Goodfellow, Bengio and Courville shows that without a non-linear function between the layers, the network "would remain a linear function of its input", however many layers it has.

The book's first example is XOR: the output is 1 when exactly one of two inputs is 1. A webshop rule such as "check the order if exactly one of the two addresses is in the Netherlands" has that shape. No straight line separates those cases, so a linear model cannot learn the rule, while a network with one hidden layer of two ReLU units can. With enough units, one hidden layer can in theory approximate almost any function (the universal approximation theorem), though the book warns that training may not find it.

Which activation functions are common?

• ReLU (rectified linear unit): max(0, x). Negative values become 0, positive values pass unchanged. The Deep Learning book calls it "the default recommendation" in modern neural networks.
• Sigmoid: 1 / (1 + e^(−x)), which squeezes any number into a value between 0 and 1. Before ReLU, most networks used sigmoid or tanh. Sigmoid still fits the output layer when the answer is the probability of a yes.
• Tanh: squeezes a number into a value between −1 and 1. According to the book it usually trains better than sigmoid when an S-shaped function is needed.
• GELU (Gaussian error linear unit): x · Φ(x), where Φ is the cumulative distribution function of the standard normal distribution. Hendrycks and Gimpel (2016) describe it as "a way to smooth a ReLU". OpenAI's first GPT (2018) and Google's BERT (2018) use it; the original Transformer of 2017 still used ReLU.
• SiLU or Swish: x · σ(x), with σ the sigmoid. Shazeer (2020) combined it with an extra gate into SwiGLU, and Meta's LLaMA (2023) uses SwiGLU in place of ReLU.
• Softmax: turns a list of scores into probabilities that add up to 1, such as the chance of each possible next token in a language model. In a transformer it also turns the scores of the attention mechanism into weights.

Where does it go wrong?

Training adjusts the weights with backpropagation, guided by the gradient: how much the error changes when a weight changes. Sigmoid and tanh flatten out for large inputs, so the gradient becomes tiny and learning stalls. That is why the book discourages sigmoid in hidden layers. ReLU has a gradient of exactly 0 for negative inputs, and a unit that outputs 0 for every input stops learning: the "dying ReLU" that Lu et al. (2019) analyse.

GELU and SiLU cost more computation than ReLU. Hendrycks and Gimpel offer faster approximations, such as x · σ(1.702x).

A better activation function gives a measurable but small gain. In Shazeer's tests the log-perplexity (lower is better) after full training fell from 1.677 with ReLU to 1.636 with SwiGLU. He offers no explanation for why the variants work beyond "divine benevolence".

Example

Suppose one layer produces three scores: −2, 0.5 and 3.

• ReLU gives 0, 0.5 and 3.
• Sigmoid gives about 0.12, 0.62 and 0.95.
• GELU gives about −0.05, 0.35 and 3.00: close to ReLU, but small negative values survive.
• Softmax gives about 0.006, 0.075 and 0.918. As the last step of a classifier, that is 92% for the third option.