In-context learning (ICL)

In-context learning is the ability of an AI model to pick up a task from the instructions and examples in your message, without being retrained. Show a chatbot three customer emails, each with the right category next to it, and it sorts a fourth email the same way. The model itself does not change: once the conversation is over, the examples are gone.

Where does the term come from?

Brown et al. (OpenAI, 2020) defined the term in the GPT-3 paper. They used it for what a large language model does within a single pass over the text: it recognises the task from the prompt and carries it out, with no gradient updates or fine-tuning. They tested three settings: zero-shot, with only a description of the task; one-shot, with one example; and few-shot, with as many examples as fit in the context window, typically 10 to 100. Larger models made better use of the examples. The survey by Dong et al. describes in-context learning as predictions based on a context with a few examples added. Few-shot prompting and zero-shot prompting are the everyday use of the same ability.

How does it work?

Nobody knows exactly. Two explanations have been studied most:

• Induction heads. Researchers at Anthropic (Olsson et al., 2022) describe parts of the attention mechanism that complete patterns: if token A was followed by token B earlier in the text, they predict B after the next A. These heads appear during training at the same moment that in-context learning improves sharply. The evidence is causal for small models and correlational for large ones.
• Hidden gradient descent. Dai et al. and von Oswald et al. (2022) argue that attention can act like a small training step on the examples. Shen et al. (2023) compared the two in a real pretrained model, LLaMA-7B, found that they behave differently, and call the equivalence an open hypothesis.

What the examples teach is also surprising. Min et al. (2022) replaced the correct answers in the examples with random ones, and performance on classification and multiple-choice tasks barely dropped, across 12 models including GPT-3. What mattered was that the examples showed the possible answers, the kind of input and the format.

Where does it go wrong?

• Choice and order of examples. Zhao et al. (2021) showed that the prompt format, the chosen examples and even their order can move GPT-3's accuracy from near chance to near the best published results. Models favour answers that appear near the end of the prompt or often in their training data. Lu et al. (2021) found that an order that works well for one model does not carry over to another.
• Length and cost. Examples take up room in the context window and are paid for as tokens with every request. Agarwal et al. (2024) tested hundreds to thousands of examples in long context windows: results kept improving on many tasks, in some cases to the level of fine-tuning, while the cost per request grew in line with the number of examples.
• Nothing is stored. Every new conversation needs the examples again; prompt caching makes that repeated part cheaper.

Example

Suppose a garden centre wants its webshop assistant to label incoming questions as "order", "return", "plant care" or "other". The prompt lists the four labels and a few real questions with the right label, and the model labels new questions without any training. Before relying on it, the shop checks a batch of real questions by hand, tries the examples in a different order, and adds an example for each mix-up it finds, such as a return request for a plant that died.