The transformer is the neural network architecture that underlies every modern Large Language Model. Introduced in the 2017 paper "Attention Is All You Need," the transformer replaced previous sequential architectures (RNNs, LSTMs) with a parallel attention mechanism that can process all tokens in a sequence simultaneously while learning which tokens are most relevant to each other. This breakthrough enabled training on vastly larger datasets and longer sequences, giving rise to GPT, Claude, Llama, and every other modern LLM. Understanding the transformer architecture explains both the capabilities and the fundamental cost structure of LLMs — why longer prompts cost quadratically more, why context windows have limits, and why these models are so effective at understanding language.
Why it matters
The transformer architecture determines the performance characteristics of every LLM you use. Its quadratic attention cost (processing 2× the tokens requires 4× the computation) directly explains why API pricing scales with token count and why context window management is critical. The architecture's parallel processing capability is what enables models to be trained on trillions of tokens — something that would take centuries with sequential architectures. For practitioners, understanding transformers provides intuition about model behavior: why LLMs excel at tasks requiring contextual understanding (each token attends to every other token), why they struggle with long mathematical computations (attention becomes diluted over very long sequences), and why prompt engineering works (the model uses attention to find the most relevant instructions in your prompt).
How it works
A transformer processes input through stacked layers, each containing two main components: a multi-head attention mechanism and a feed-forward neural network. In the attention step, every token computes a relevance score with every other token (self-attention), allowing the model to understand that in "The bank approved the loan," the word "bank" is strongly associated with "approved" and "loan" (financial context) rather than with "river" or "shore." Multiple attention heads run in parallel, each learning different types of relationships — syntactic structure, semantic meaning, positional patterns. The feed-forward network then transforms the attention outputs. Residual connections and layer normalization stabilize training across dozens or hundreds of layers. For text generation, a causal mask ensures the model only attends to previous tokens, preventing it from "looking ahead" — the model generates strictly left-to-right, one token at a time.
Example
Consider how a transformer handles a translation prompt: "Translate to Dutch: The bank by the river was steep." The attention mechanism first resolves the ambiguity of "bank" — attention heads note strong connections between "bank," "river," and "steep," correctly identifying this as a riverbank rather than a financial institution. Other heads track the instruction to translate, maintaining awareness across the entire sequence. The feed-forward layers encode the transformation from English to Dutch linguistic patterns. The model generates "De oever bij de rivier was steil" — correctly choosing "oever" (riverbank) rather than "bank" (financial bank). This contextual disambiguation across the full input, processed in parallel rather than word-by-word, is the transformer's defining advantage over all previous architectures.