Real-world agent reliability gap

The real-world agent reliability gap is the difference between how well AI agents score on popular tests and how reliably they handle real work. An agent can pass a benchmark and still handle the same customer request correctly one day and wrongly the next. In May 2026, two tests built from real work tasks found that even the best agents scored below 50%, so their work still needs checking.

What do researchers mean by it?

The name describes a problem that researchers measure in several ways. Rabanser, Kapoor and colleagues at Princeton (Towards a Science of AI Agent Reliability, February 2026) start from the observation that rising benchmark scores suggest rapid progress while many AI agents still fail in practice. They argue that a single success rate hides four properties: whether an agent behaves consistently across runs, withstands small changes, fails predictably and keeps the damage of its errors limited. Across 15 models released over 24 months, reliability improved only a little while accuracy rose.

Why do benchmark scores overstate reliability?

Popular tests are close to their ceiling. On OpenAI's GDPval, GPT-5.4 matched or beat experts in 83.0% of comparisons, which the authors of JobBench call close to saturation. On JobBench itself (25 May 2026), 130 tasks that workers most want to hand off, the best agent scored 45.9% (Claude Opus 4.7 in Claude Code). On ITBench-AA (27 May 2026), 59 Kubernetes incidents from Artificial Analysis and IBM, Claude Opus 4.7 led with 47%, ahead of GPT-5.5 with 46%; the best open-weights model reached 40%.

A single run hides inconsistency. τ-bench (Yao et al., June 2024) simulates customer service for a retailer and an airline, and introduced pass^k: the chance that an agent solves the same task in all of k attempts. GPT-4o solved fewer than half of the tasks, and its pass^8 on the retail tasks was below 25%. Rabanser et al. saw the same pattern in 2026: agents that can solve a task often fail to solve it every time, and many react to a rephrased instruction, while they handled injected technical faults well.

Some scores flatter the agent. Kapoor et al. (AI Agents That Matter, July 2024) found that many agent benchmarks have inadequate or no hold-out sets, so agents take shortcuts and overfit to the test. The setup counts as well: on JobBench, Claude Sonnet 4.6 scored 36.9% in Claude Code and 30.6% in OpenClaw, so a score describes one combination of model and agent software.

What does this mean if you use agents?

Test the exact model and agent software you plan to use, on tasks like your own, and run each task more than once: the share of tasks an agent gets right every time says more than one success rate (agent evaluation). For actions that are hard to undo, keep a person in the loop (human-in-the-loop). The scores do move. By September 2026 the top score on the JobBench leaderboard was 65.7% (Claude Opus 5), and on the ITBench-AA leaderboard 56.2% (GPT-5.6 Sol). The post Why do AI agents lag behind the hype on real work? covers both May 2026 benchmarks in detail.

Example

Suppose a webshop wants an agent to handle return requests. It collects 20 typical requests from its inbox and lets the agent handle each one five times. The requests the agent gets right in all five runs are the ones it can take over on its own. The rest need a staff member's check, however good the agent's score on a public benchmark is. That is the pass^k idea from τ-bench, applied to the shop's own work.