Token in AI

A token is a small piece of text that an AI model reads or writes: a short word, part of a word, or a punctuation mark. The AI services behind chatbots and other apps bill per token, so a webshop pays for the text its chatbot sends to the model and for every answer that comes back. In English, 100 tokens is about 75 words; Dutch usually needs more.

What does a token look like?

Common words are usually a single token, and rare or long words are split into pieces (Hugging Face). In OpenAI's Tokenizer, set to its GPT-5.x models in September 2026, "the" is one token, "understanding" on its own is two ("under" and "standing"), and the Dutch word "klantenservicemedewerker" is six. Capitals and a leading space change the split too: "red", "Red" and " red" can be different tokens (OpenAI). Images and audio become tokens too: in September 2026, Google counts an image of up to 384 by 384 pixels as 258 tokens for Gemini, and a second of audio as 32.

How many words is a token?

For English text, OpenAI puts a token at about 4 characters or three-quarters of a word. Google gives about 4 characters for Gemini, with 100 tokens equal to 60 to 80 English words. These are averages, and every model family uses its own tokenizer. Anthropic's Claude models from version 4.7 onward use a newer tokenizer that turns the same text into about 30% more tokens than earlier Claude models, so Anthropic advises counting again for each model (Anthropic).

The language matters too. Petrov et al. (2023) tokenized the same 2,000 sentences in 200 languages and found differences of up to 15 times. With the tokenizer that GPT-4 and ChatGPT used at the time, Dutch needed about 1.6 times as many tokens as English for the same text, which made it about 1.6 times as expensive.

What does a token cost?

Providers price input tokens, the text you send, separately from output tokens, the text the model writes. In September 2026, Anthropic charges $1 per million input tokens and $5 per million output tokens for Claude Haiku 4.5, and $2 and $10 for Claude Sonnet 5 (pricing). Reasoning models also write hidden reasoning tokens, which OpenAI bills as output tokens, so a short answer can use more tokens than its visible text suggests. OpenAI therefore advises comparing the total cost of a task instead of the price per million tokens, because models split the same text differently and write different amounts. Limits such as the context window are counted in tokens too, and token economics covers pricing in more depth.

How does a model use tokens?

The tokenizer splits the text into tokens from a fixed list, the vocabulary, and gives each token a number. The model turns each number into a list of numbers learned during training, an embedding (Vaswani et al., 2017). A large language model then writes its answer one token at a time (autoregressive generation).

Example

Suppose a webshop's chatbot handles 1,000 conversations a day, each with about 2,000 input tokens (instructions, product information and the customer's messages) and 300 output tokens. That makes 2 million input tokens and 300,000 output tokens a day. On Claude Haiku 4.5, at the September 2026 prices, the day costs $2 plus $1.50, or $3.50. On Claude Sonnet 5 each token costs twice as much, and its newer tokenizer makes about 30% more tokens from the same input, so the input alone rises from $2 to about $5.20. The instructions repeat in every conversation, so prompt caching helps: for both models, text read from the cache costs a tenth of the input price.