Encoder-decoder architecture

An encoder-decoder is a design for AI models with two halves. The encoder reads the whole input, for example an English sentence, and turns it into numbers that capture its meaning. The decoder uses those numbers to write the output, such as the Dutch translation, one word at a time. Translation systems were built this way; most of the language models behind chatbots use only the second half.

Where does it come from?

In 2014, two groups built neural networks for translation from an encoder and a decoder. Cho et al. called theirs an RNN Encoder-Decoder. Sutskever, Vinyals and Le used one LSTM to turn an English sentence into a vector of fixed size and another to decode the French translation from it. Bahdanau, Cho and Bengio saw that fixed-size vector as a bottleneck and let the decoder search the source sentence for the parts that matter for each word: an early form of the attention mechanism. In 2016 Google Translate switched to GNMT, an encoder-decoder with 8 encoder and 8 decoder layers and attention.

How does the transformer version work?

The original transformer (Vaswani et al., 2017) was an encoder-decoder for translation, with a stack of six layers in each half:

• Encoder: each position can attend to every position in the input, so it reads the whole sentence in both directions.
• Decoder: generates the output one token at a time and may only look at the tokens it has already produced. That is autoregressive generation.
• Cross-attention, which the paper calls encoder-decoder attention: the decoder's queries look up keys and values from the encoder's output, so every output step can use every part of the input.

What are the three variants?

Later models kept one half or both (Hugging Face LLM course):

• Encoder-only, such as BERT (2018): suited to understanding a whole text, for classification or finding names in a sentence.
• Decoder-only, such as GPT and Llama: suited to generating text. Most modern large language models use this design.
• Encoder-decoder, such as T5 (2019) and BART: suited to turning one text into another, such as translating or summarising.

Is the encoder-decoder still used?

Yes, and the two halves need not be alike. In 2020 Google Translate combined a transformer encoder with a faster RNN decoder, because most of the quality gain came from the encoder (Google Research, June 2020). In July 2025 Google turned decoder-only Gemma 2 models into encoder-decoders, T5Gemma. It can pair a large encoder with a small decoder, such as 9B with 2B, for tasks like summarising, where understanding the input matters most. Google reports results comparable to or better than the decoder-only originals. In September 2026 DeepSeek released V4.1-Flash, a language model with what it calls a causal encoder-decoder architecture (see DeepSeek).

Example

Suppose a webshop wants its product descriptions in German. An encoder-decoder model first reads the whole Dutch description. The decoder then writes the German text word by word, and for each word looks back at the relevant part of the Dutch text through cross-attention. A decoder-only chatbot does the same job by reading the Dutch text as part of its prompt and continuing with the translation.