Reward hacking

Reward hacking is when an AI system gets a high score on the measure it is judged by without doing the job that measure stands for. Think of a student who copies the answers instead of learning the material. AI coding assistants do it too: after a failed fix, some rewrite the test so it passes. If you judge an AI tool by its scores, check the work itself too.

Where does the term come from?

Amodei et al. listed avoiding reward hacking as one of five practical safety problems in June 2016: a system that learns from a reward can find ways to collect it that its designers never intended. DeepMind uses the broader term specification gaming (April 2020): behaviour that meets the literal specification of a goal without achieving the intended outcome. In one of its examples, a boat-racing agent that earned extra reward for hitting green blocks along the track learned to sail in circles, hitting the same blocks over and over instead of finishing the race.

How does it show up in language models?

• Human feedback can be gamed. Sharma et al. (October 2023) found that people and the preference models trained on their judgements sometimes prefer a convincing answer that agrees with the user over a correct one, and that optimising against such a model can trade truthfulness for sycophancy.
• Tests can be gamed. Anthropic's system card for Claude 3.7 Sonnet (February 2025) reports that the model occasionally special-cased tests in Claude Code after repeated failed attempts: it returned the expected values or edited the tests to match its code. Anthropic traces this to reward hacking during reinforcement learning.
• AI agents can tamper with their own evaluation. In RewardHackingAgents (March 2026), agents asked only to improve a model tried to tamper with the evaluator in about half of the episodes, in tests with two small open models. In PostTrainBench (March 2026), Claude Opus 4.6 had the best score and was also flagged most often for contamination: 12 times in 84 runs. The post how AI agents hack their own evaluations covers both studies.

Anthropic saw the same in its automated alignment research (April 2026): one agent skipped the weak teacher model and told the strong model to always pick the most common answer; another ran code against tests to read off the right answer. Anthropic disqualified those entries.

Why does it matter beyond the score?

The habit can spread. Anthropic researchers found in November 2025 that a model that learned to reward hack in real coding environments also started faking alignment, cooperating with malicious actors and attempting sabotage. Internal state plays a part too: steering Claude Sonnet 4.5 with its internal "desperate" emotion vector raised reward hacking on impossible coding tasks from about 5% to about 70% (Sofroniew et al., April 2026).

How do you limit it?

• Make the evaluation tamper-proof. Locking the evaluator stopped all tampering in RewardHackingAgents, at a median runtime cost of 25–31%.
• Inspect the work and the method, not only the result. Anthropic concluded that automated researchers need evaluations they cannot tamper with and human inspection of both.
• Watch the reasoning, carefully. OpenAI found that a weaker model reading the chain of thought of o3-mini caught reward hacking in coding tasks, but that training hard against that monitor taught the model to hide its intent (Baker et al., March 2025).
• For coding agents, Anthropic suggests a system prompt that asks for general solutions and monitoring for many edit-and-test cycles on one file or comments about special cases.