Tokenizer

A tokenizer is the part of an AI system that cuts text into tokens, the small pieces a language model works with, and gives each piece a number. When the model answers, the tokenizer turns those numbers back into text. Each model family has its own, so the same text can come to a different token count, and a different price, per model.

How does a tokenizer work?

A tokenizer first learns a vocabulary from a large collection of text, and then uses that vocabulary to split new text. The usual method in current large language models is byte pair encoding (BPE) or a variant of it (Singh and Strouse, 2024). Sennrich et al. (2015) adapted this compression algorithm from 1994 to split words: start with single characters, find the pair of neighbouring symbols that occurs most often, merge it into a new symbol, and repeat until the vocabulary has the chosen size.

Hugging Face shows the steps on the words "hug", "pug", "pun", "bun" and "hugs": "u" and "g" merge first into "ug", then "u" and "n" into "un". Frequent words end up as one token; rare words stay in pieces. GPT-2 uses byte-level BPE, which starts from the 256 possible byte values instead of characters, so it can encode any text.

There are variants. WordPiece, which Google's BERT uses with a vocabulary of 30,000 tokens, picks the merge that most increases the likelihood of the training data instead of the most frequent pair. Unigram works the other way round, starting with many candidates and removing the least useful tokens. Google's SentencePiece library (Kudo and Richardson, 2018) runs BPE or Unigram on raw text and treats the space as an ordinary symbol, written "▁", so it also handles languages without spaces between words, such as Japanese. Meta's Llama 2 used its BPE with 32,000 tokens (Touvron et al., 2023).

How big is the vocabulary?

GPT-2 had 50,257 tokens. In OpenAI's tiktoken library, cl100k_base, the encoding of GPT-4, has about 100,000, and o200k_base, used for GPT-4o and GPT-5, about 200,000. Meta's Llama 3 uses 128,000: 100,000 taken from tiktoken plus 28,000 for languages other than English. On English text, that raised the average from 3.17 characters per token with Llama 2's tokenizer to 3.94, so the model reads more text for the same training compute (Llama 3 paper). A larger vocabulary splits text into fewer tokens, but the risk grows that rare tokens get too little training (Tao et al., 2024).

Where does it go wrong?

• Other languages need more pieces. In o200k_base, " fascinating" is one token and the Dutch " fascinerend" three (" fasc", "iner", "end"). The older cl100k_base cuts "klantenservicemedewerker" into nine pieces, o200k_base into six.
• Numbers are split in odd places. GPT-4's tokenizer has a token for every number of one to three digits and splits longer numbers from the left, so "1234567" becomes "123", "456" and "7"; Llama and PaLM split numbers into single digits. When Singh and Strouse added commas every three digits from the right, GPT-3.5 (March 2023) solved 97.8% of additions of 7- to 9-digit numbers, against 75.6% without commas, both with eight worked examples in the prompt.
• Some tokens are in the vocabulary but barely occur in the training data. These under-trained tokens, such as "SolidGoldMagikarp", can make a model behave strangely; Land and Bartolo (Cohere, 2024) found them across many models.
• Not every tokenizer is public. Anthropic's tokenizer package says it is no longer accurate from Claude 3 onward; for Claude, the API counts the tokens.

Example

This snippet splits two sentences with tiktoken's o200k_base; the comments show the output:

1
import tiktoken
2
 
3
enc = tiktoken.get_encoding("o200k_base")  # the encoding of GPT-4o and GPT-5
4
for text in ["Tokenization is important", "De klantenservicemedewerker belt je morgen."]:
5
    ids = enc.encode(text)
6
    print(len(ids), [enc.decode([i]) for i in ids])
7
# 4 ['Token', 'ization', ' is', ' important']
8
# 10 ['De', ' klanten', 'serv', 'ic', 'emed', 'ewerker', ' belt', ' je', ' morgen', '.']

Decoding the numbers gives back exactly the original sentence: tiktoken's documentation calls BPE reversible and lossless.