Top-p (nucleus) sampling

Top-p sampling, also called nucleus sampling, is a setting that limits which words an AI model can choose as it writes. At top-p 0.9, the model only picks from its likeliest next words that together make up 90% of the probability, so it skips the long tail of unlikely words. Lower values give more predictable text, higher values more varied text. OpenAI describes it as an alternative to temperature.

How does it work?

Holtzman et al. introduced nucleus sampling in 2019. A large language model writes one token at a time, and at each step it gives every possible next token a probability. Top-p sorts the tokens from most to least likely, keeps the smallest group whose probabilities add up to at least p, scales those probabilities up so they sum to 1 again, and draws the next token from that group. Google's documentation gives an example: if tokens A, B and C have probabilities of 0.3, 0.2 and 0.1 and top-p is 0.5, the model picks A or B and never C.

The size of the group adapts. According to Holtzman et al., this nucleus ranges from one to a thousand candidate tokens. Top-k sampling, by contrast, always keeps a fixed number k: when a few tokens hold almost all the probability, a large k lets in unlikely ones, and when many words fit, a small k cuts off good options.

The order with temperature differs per system. Hugging Face's Transformers library applies temperature first and then cuts off with top-k and top-p (source code). Google describes Gemini the other way round: top-k, then top-p, then temperature for the final draw.

Does it make the text better?

For long, open-ended text it did in Holtzman et al.'s tests with GPT-2 Large. With p = 0.95, nucleus sampling came closest to human text in perplexity and scored highest on HUSE, a combined human and statistical evaluation. Methods that always prefer the likeliest words, such as greedy decoding and beam search, got stuck repeating themselves, and pure sampling drifted into incoherent text.

More variety also means more room for mistakes. Lee et al. (2022) counted names and other named entities in generated text that do not appear in the reference source, a form of hallucination. For their 530-billion-parameter model, 33.3% of these entities were wrong with top-p at 0.9, against 20.9% with greedy decoding, which in turn ended in repetition far more often (15.9% of texts against 0.1%). Their factual-nucleus sampling lowers p step by step within each sentence and resets it at the next one. Min-p sampling (Nguyen et al., 2024) instead sets the cut-off relative to the likeliest token.

Which values do providers allow?

In September 2026, OpenAI's API reference explains that 0.1 means only the tokens in the top 10% of the probability mass are considered, and recommends changing top_p or temperature but not both. Google gives a default top-p of 0.95 for Gemini. Anthropic's API reference marks top_p as deprecated: models released after Claude Opus 4.6 only accept values of 0.99 or higher and reject others with an error. Hugging Face Transformers uses greedy search by default, with top_p at 1.0.

Example

Suppose a shop's chatbot continues the sentence "We are open from Monday to". The model gives "Friday" a probability of 0.6, "Saturday" 0.3 and "Sunday" 0.05, and thousands of other tokens share the rest. At top-p 0.9 only "Friday" and "Saturday" remain, because 0.6 plus 0.3 reaches 0.9; at top-p 1, an odd word keeps a small chance. Either way the model can still pick "Saturday" while the shop closes on Fridays. Top-p controls variety, so the real opening hours have to come from the shop's own data, for example through RAG.