Agent evaluation

Agent evaluation means testing whether an AI agent really does the job before you let it work for you. An AI agent is software that carries out tasks on its own, such as looking up an order and issuing a refund. The test gives it realistic tasks and checks the result: was the refund actually processed, or did the agent only say so?

How does it work?

In Anthropic's guide Demystifying evals for AI agents (January 2026), an eval is a task with clear success criteria, plus graders that score each attempt. Each attempt leaves a transcript: the agent's messages, its tool calls (actions through software, such as a search or a database update) and the intermediate results. The guide separates this from the outcome, the final state. A flight-booking agent can say "Your flight has been booked", while the outcome is whether a reservation exists in the database.

Graders come in three kinds, often combined:

• Code checks the final state, runs tests or verifies which tools were called. It is fast and reproducible, and rejects valid answers that differ from the expected pattern.
• A language model as judge (LLM-as-a-judge) scores against a rubric. It handles open-ended answers, can vary between runs and needs calibration against human graders.
• Human graders are the most reliable and the slowest.

For a conversational agent, a second language model often plays the customer. In τ-bench (Yao et al., 2024), a simulated customer talks to a retail or airline agent, and the benchmark then compares the database with the expected state. Anthropic suggests starting with 20 to 50 simple tasks drawn from real failures.

Why is one successful run not enough?

Agents do not behave the same way on every run. τ-bench therefore introduced pass^k, the chance that an agent solves a task in all k attempts. GPT-4o with function calling solved about 61% of the retail tasks in one attempt, and about 25% in all eight attempts. For an agent that serves customers, that consistency is what counts.

Example

IBM Research's VAKRA (Inside VAKRA, April 2026) gives agents more than 8,000 locally hosted APIs on real databases in 62 domains, plus document collections, with tasks that can take 3 to 7 steps. Grading runs as a waterfall. A program first checks any rules for tool use. The benchmark then reruns the agent's tool calls and checks whether their results hold all the information of the reference solution, so another valid route also passes. Where that check is inconclusive, and for the final answer, a language model judges.

Ecom-RLVE (April 2026) uses no model judge. In eight webshop environments, such as cart building and returns, a program that knows the hidden goal scores whether the task was done, how many turns the agent wasted through its own mistakes, and whether it recommended products it never looked up. One difficulty setting controls 12 aspects of a task, such as how often the simulated customer leaves out a requirement. Because code computes the score, the environments can also serve as rewards for training with reinforcement learning with verifiable rewards (RLVR).

Where does it go wrong?

• The grader can be wrong. Anthropic reports that Claude Opus 4.5 first scored 42% on CORE-Bench. After fixes to grading that, for example, rejected 96.12 where 96.124991… was expected, and with a less restrictive setup around the model, it scored 95%.
• Checking the exact order of tool calls is brittle, because agents find valid routes nobody foresaw. Anthropic advises grading what the agent produced over the path it took.
• The end state does not show everything. The τ-bench authors note that an agent can process a correct return without the customer's explicit confirmation, which breaks the policy.
• Agents can exploit loopholes in the test, a form of reward hacking.