Why do AI agents lag behind the hype on real work?

Partly because older tests of work tasks are close to saturation, while tests built from real work are not. In late May 2026, two new benchmarks measured AI agents on work that professionals actually do: JobBench, 130 tasks that workers most want to hand off, and ITBench-AA, 59 Kubernetes incidents of the kind site reliability engineers handle. On both, the best agent scored below 50%. On OpenAI's GDPval, an older test of work tasks, GPT-5.4 already matched or beat experts in 83.0% of comparisons.

What does JobBench measure?

JobBench comes from researchers at the University of Washington and other institutions and was published on 25 May 2026 (Li et al.). It starts from WorkBank, a survey in which more than 1,500 workers rated, duty by duty, whether they would like an AI agent to take over parts of their job. The authors picked 35 occupations from it, among them reporters, customer service representatives, bookkeeping clerks, online merchants and lawyers, and built 130 tasks: a main set of 65 and an easier set of 65.

Each task is a folder with an instruction, the deliverables to produce and on average 3.9 reference files, such as CSV, PDF and Excel files, which can contradict each other. In the paper's reporter task, the agent cross-checks water-quality data, EPA guidance and health surveillance data across several years before it assembles an editorial package. The grading uses chains of yes-or-no checks, 35.6 per task on average. A chain only earns points when every check in it passes, so the right fact reached through a wrong inference earns nothing. A language model grades the work: Grok 4.1 Fast, checked against Claude Opus 4.5.

How did the agents score on JobBench?

The authors tested 36 combinations of model and agent software (Claude Code, Codex CLI, OpenCode and OpenClaw). On the main set, Claude Opus 4.7 in Claude Code scored 45.9%, GPT-5.5 in Codex CLI 42.7% and GPT-5.4 in Codex CLI 38.9%. Outside the Claude and GPT families, no combination scored above 19%. Google's entries were Gemini 3 Pro and Gemini 3 Flash, both run in OpenCode, at 11.4% each. At list prices, a full run of the main set cost $210 with Opus 4.7 and $44 with GPT-5.5.

The agent software matters as well: Claude Sonnet 4.6 scored 36.9% in Claude Code and 30.6% in OpenClaw.

What does ITBench-AA measure?

ITBench-AA comes from Artificial Analysis, an independent company that benchmarks AI models, and IBM, and was published on 27 May 2026. It builds on IBM's ITBench (2025) and starts with site reliability engineering (SRE), the work of keeping production systems running. Tasks on cloud costs (FinOps) and security (CISO) are to follow.

Each of the 59 tasks (40 public, 19 kept private) is a snapshot of a Kubernetes incident, with alerts, events, traces, metrics, logs and the application topology. The agent investigates it with shell commands and must name the smallest set of Kubernetes objects that caused the incident. Faults include exhausted resource quotas, failed rollouts, exhausted connection pools and network partitions. In one public task, a network policy blocks all traffic to the frontend. Every model runs in the same agent software, the open-source Stirrup harness of Artificial Analysis, with at most 100 turns and three attempts per task. The scoring is strict: an attempt that misses any real cause scores zero, and one that finds them all scores the share of named objects that really are causes.

How did the agents score on ITBench-AA?

Claude Opus 4.7 led with 47%, ahead of GPT-5.5 with 46% and Qwen3.7 Max with 42%. The best open-weights model, GLM-5.1, reached 40%, level with Gemini 3.5 Flash, and Gemini 3.1 Pro Preview scored 30%. Working longer did not help. GPT-5.5 averaged 31 turns per task, Gemini 3.1 Pro Preview 83. Models that kept investigating named extra objects, such as the tool that injected the fault or symptoms that occurred at the same time, and those count as false positives. Costs varied widely: Gemma 4 31B scored 37% for $0.14 per task, while Opus 4.7 cost $5.38 per task.

Why do these scores lag behind the hype?

Both benchmarks differ from the popular tests in the same ways:

• The popular tests are easier for these models. On GDPval, GPT-5.4 matches or beats experts in 83.0% of comparisons, which the JobBench authors call close to saturation, and according to Artificial Analysis, frontier models score considerably higher on Terminal-Bench than on ITBench-AA.
• The tasks ask the agent to reconcile sources: files that contradict each other in JobBench, a full incident snapshot with many symptoms in ITBench-AA.
• Half-right work earns nothing. A JobBench chain fails on one wrong step, and an ITBench-AA attempt that misses one cause scores zero.
• The setup counts. The same model scores differently in different agent software, and more turns do not produce better diagnoses.

What does this mean if you build with agents?

On work like this, an agent's output still needs checking. The JobBench authors conclude that agents "continue to make substantial errors" on the professional reasoning experts most want to hand off. Test the exact combination of model and agent software you plan to use, on tasks like your own, because a benchmark score describes one setup. For agent evaluation, both benchmarks show a way to grade strictly: fixed checks per task, and no credit for a lucky answer.

By September 2026, both leaderboards showed higher top scores: 65.7% for Claude Opus 5 on the JobBench leaderboard and 56.2% for GPT-5.6 Sol on the ITBench-AA leaderboard.