Transformer

A transformer is the type of AI model behind ChatGPT, Gemini and most other chatbots that write text. It reads your whole question at once and works out which words belong together, for example that "it" in "the letter came late because it went to the wrong address" means the letter. Then it writes its answer piece by piece. The T in GPT, and so in ChatGPT, stands for transformer.

Where does it come from?

Eight researchers working at Google introduced the transformer in 2017 in the paper Attention Is All You Need (Vaswani et al.). The language models before it, recurrent networks such as LSTMs, read a text word by word, and the paper notes that this sequential nature prevents parallel work within a training example. The transformer drops the word-by-word reading and relies on attention alone, so it can process all words of a sentence at the same time. The first transformer was a translation model. Its large version, trained for 3.5 days on eight GPUs, scored 28.4 BLEU (a measure of how closely a translation matches human ones) on the WMT 2014 English-German test, more than 2 points above the best earlier results.

How does it work?

A transformer for text works in four steps:

1. The text is split into tokens, words or parts of words, and each token becomes a list of numbers, an embedding. Because the model sees all tokens at once, a positional encoding adds where each token stands.
2. The embeddings pass through a stack of identical layers. In each layer, self-attention lets every token take in information from the tokens that matter to it, and a feed-forward network then processes each token on its own. Both parts have a residual connection and layer normalization around them.
3. After the last layer, the model turns the result into a probability for every token in its vocabulary.
4. To write text, the model picks a likely next token, adds it to the input and runs again. A mask makes sure each token only looks at the tokens before it (autoregressive generation).

The original had 6 layers in its encoder and 6 in its decoder, and 65 million parameters in its base version. Meta's Llama 3 405B (2024) has 126 layers and 405 billion parameters.

Which kinds are there?

The first transformer had two halves: an encoder that reads the input and a decoder that writes the output (encoder-decoder). Later models often keep one half:

• Encoder-only, such as BERT (Google, 2018), which reads a text in both directions and suits tasks such as classification.
• Decoder-only, such as GPT and Llama, which writes text token by token. According to their technical reports, GPT-4 and Gemini are built on transformers too.
• Encoder-decoder, such as the original, which turns one text into another, for example a translation.

Where does it fall short?

Attention compares each token with every other token, so that work grows with the square of the text length: twice as many tokens means four times as many pairs. While generating, the model also keeps information about all earlier tokens in its KV cache, which grows with every token. The Mamba-3 paper sums this up as "quadratic compute and linear memory". A transformer can also only take in a limited amount of text at once, its context window: up to 128K tokens for Llama 3. State space models are an alternative with a memory of fixed size, and some newer models combine both.

Example

Suppose a customer types "Can I still return my order?" into a webshop's chatbot. The transformer splits the question into tokens and turns each into an embedding. In every layer, each token takes in information from the others, so "return" is read in the context of "order". After the last layer, the model calculates which token most likely starts the answer, adds it and repeats until the answer is complete. If the shop's returns policy is not in the prompt, for example through RAG, the model can only write a plausible general answer.