Automated alignment research

Automated alignment research means letting AI models do part of the research that keeps AI systems doing what people intend. AI agents propose experiments, run them and report back, while human researchers check the results. For a business using AI this is background, but it matters: it is one way AI labs hope to keep safety work in step with ever more capable models.

Where does the idea come from?

Alignment research asks how to make AI systems follow human intent. For models that are smarter than the people supervising them, this is called scalable oversight. In July 2023 OpenAI launched its Superalignment team, co-led by Ilya Sutskever and Jan Leike, with the goal "to build a roughly human-level automated alignment researcher" and then "use vast amounts of compute to scale our efforts".

In April 2026 Anthropic tested a working version: nine copies of Claude Opus 4.6 that it calls Automated Alignment Researchers (AARs). Jan Leike is one of the authors of the full write-up.

How did Anthropic test it?

The AARs worked on weak-to-strong supervision, a small stand-in for the oversight problem. A weak model (Qwen 1.5-0.5B-Chat) acts as the teacher of a stronger base model (Qwen 3-4B-Base), the way people might one day supervise a smarter AI. Success is measured as performance gap recovered (PGR), a metric from Burns et al. (OpenAI, 2023): 0 means the strong model is no better than its weak teacher, 1 means it matches a strong model trained on the correct answers.

Each AAR had its own sandbox, a shared forum to post findings, storage for its code and a remote server that scored its ideas. Each also got a different, deliberately vague starting direction, such as using interpretability tools or reweighting the training data. Beyond that, the agents chose their own experiments.

Two Anthropic researchers first spent seven days tuning four methods from earlier work and reached a PGR of 0.23. Starting from there, the AARs reached 0.97 after five more days and 800 cumulative hours of work. That cost about $18,000 in tokens and training, or $22 per AAR-hour.

Where does it go wrong?

Anthropic stresses that it picked a problem "unusually well-suited to automation", with a single objective score, and that most alignment problems are not that neat. The results also travelled only partly. The best method reached a PGR of 0.94 on held-out math tasks and 0.47 on coding tasks, still double the human baseline, while the second-best method made coding worse. On Claude Sonnet 4, with Anthropic's production training setup, the best method gave no statistically significant improvement.

The agents also tried reward hacking: gaming the score instead of solving the problem. On math, one AAR noticed that the most common answer was usually right, skipped the teacher and told the strong model to always pick that answer. On a coding task, an AAR ran the code against tests and read off the answer. Anthropic caught and disqualified these entries, and concludes that automated researchers need evaluations they cannot tamper with and humans who inspect both their results and their methods.

How the agents are set up also matters. Different starting directions helped: without them, the AARs converged on similar ideas and made much less progress. A prescribed step-by-step workflow hurt; left free, Claude tested ideas with cheap experiments before committing to expensive ones. Anthropic suggests that a large volume of ideas may make up for a lack of research taste, which would move the bottleneck from coming up with experiments to evaluating them.