Perplexity in NLP

Perplexity is a score for how well a language model predicts text. The model reads a text piece by piece and gives a probability to each piece that actually comes next; perplexity turns those probabilities into one number. The lower it is, the less the model was surprised. Model builders use it to compare versions of a model, but a lower score does not guarantee better answers in practice.

How is it calculated?

Take a test text the model has not seen and split it into tokens. For each token, the model gives the probability of that token given everything before it. Perplexity is the exponent of the average negative logarithm of those probabilities (Hugging Face): perplexity = exp(-(log p1 + log p2 + ... + log pN) / N). That equals the exponent of the cross-entropy between the text and the model's predictions, the loss language models are trained on, so a training loss converts straight into a perplexity.

The textbook Speech and Language Processing by Jurafsky and Martin describes perplexity as a weighted average branching factor: roughly the number of equally likely options the model still has per token. A model that guesses evenly among 50,000 tokens has perplexity 50,000; a model that always predicts the right token with certainty has perplexity 1.

What do the numbers look like?

Jurafsky and Martin trained models on 38 million words of the Wall Street Journal and tested them on 1.5 million words. Perplexity fell from 962 for a model that looks at single words to 170 for word pairs and 109 for sequences of three words. On the One Billion Word Benchmark (Chelba et al., 2013), a classic 5-gram model scored 67.6, and a combination of models, with a recurrent neural network as the strongest, cut that by 35%. In cross-entropy that was a reduction of 10%: the exponent magnifies differences.

Perplexity is also a quick check after compressing a model. GPTQ (Frantar et al., 2022) reports it on WikiText-2, PTB and C4 because it reacts strongly to quantization. At 4 bits, OPT-175B lost 0.03 perplexity with GPTQ and 2.2 with plain rounding.

Where does it go wrong?

• Scores are only comparable on the same test text with the same tokenizer. Jurafsky and Martin write that perplexities are only comparable with identical vocabularies, and Hugging Face notes that tokenization directly affects the score.
• The method of calculation changes the number. Hugging Face measured GPT-2 large on WikiText-2: in separate blocks of 1,024 tokens it scored 19.44, close to the 19.93 in the GPT-2 paper, and with a sliding window that gives each token more context, 16.44.
• Test text that was in the training data makes the score artificially low (Jurafsky and Martin).
• It fits models that write left to right. For masked models such as BERT, perplexity is not well defined (Hugging Face).
• A lower perplexity does not guarantee better results on a real task, and Jurafsky and Martin advise confirming it with an end-to-end test. For long texts, Hu et al. (2024) found no correlation between perplexity and how well models understand them.

Example

Jurafsky and Martin use a language of three words: red, blue and green. A model that finds each word equally likely has perplexity 3 on any text. A model that gives red a probability of 0.8 and the other two 0.1 each scores 1.89 on the test text "red red red red blue". In Python:

1
import math
2
probs = [0.8, 0.8, 0.8, 0.8, 0.1]  # probability of each word that actually came next
3
perplexity = math.exp(-sum(math.log(p) for p in probs) / len(probs))
4
print(round(perplexity, 2))  # 1.89