ITBench-AA is a test of how well AI agents find the cause of an IT outage. Each task is a snapshot of a failure in Kubernetes, software that many companies use to run their web applications on servers. The agent searches the alerts and logs and has to name exactly what caused the problem. At the launch in May 2026, the best model scored 47%.
Who made it?
Artificial Analysis, a company that benchmarks AI models, published ITBench-AA with IBM on 27 May 2026 (announcement). It is Artificial Analysis's own implementation of IBM's ITBench from February 2025. That benchmark had 94 scenarios in three areas: site reliability engineering (SRE), which keeps production systems running; compliance and security (CISO); and cloud cost management (FinOps). In the ITBench paper, agents resolved 13.8% of the SRE scenarios, 25.2% of the CISO scenarios and none of the FinOps scenarios. ITBench-AA starts with SRE; FinOps and CISO tasks are to follow.
What does an agent have to do?
There are 59 tasks: 40 public ones and 19 new ones kept private. Each is a snapshot of a Kubernetes incident with alerts, events, traces, metrics, logs and the topology of the application. The faults include exhausted resource quotas, failed rollouts, exhausted connection pools and network partitions. The agent examines the snapshot with shell commands and submits the smallest set of Kubernetes objects, such as Deployments, Services or Pods, that caused the incident. The snapshot is offline, so the agent diagnoses and does not repair anything.
How is it scored?
Every model runs in the same open-source agent software, Artificial Analysis's Stirrup harness, with shell access to a sandbox, at most 100 turns and three attempts per task. The scoring is strict. An attempt that misses any of the real causes scores zero. One that finds them all scores the share of the named objects that really are causes. The headline score is the average over the 59 tasks and three attempts.
How did the models score?
At the launch, Claude Opus 4.7 led with 47%, ahead of GPT-5.5 with 46% and Qwen3.7 Max with 42%. The best open-weights model, GLM-5.1, reached 40%, level with Gemini 3.5 Flash, and Gemini 3.1 Pro Preview scored 30%. Working longer did not help: GPT-5.5 averaged 31 turns per task, Gemini 3.1 Pro Preview 83. Models that kept investigating named extra objects, such as the tool that injected the fault or symptoms that occurred at the same time, and those count as false positives. Costs ranged from $0.14 per task for Gemma 4 31B, which scored 37%, to $5.38 for Claude Opus 4.7. By September 2026 the leaderboard showed a higher top score: 56.2% for GPT-5.6 Sol.
What does it leave out?
- It covers only SRE: in September 2026 the leaderboard still lists only the SRE tasks.
- It measures only the diagnosis from a snapshot; repairing a live system is not part of it.
- The scores compare models inside one harness and say less about agent products with their own tools.
The post Why do AI agents lag behind the hype on real work? compares ITBench-AA with JobBench.
Example
In one public task, users see failures in the frontend of a demo application. The agent reviews the alerts to find when the incident started, narrows the failure down to frontend traffic with traces and logs, finds the affected services in the topology, and discovers in the Kubernetes manifests a network policy that blocks the frontend. The right answer is that one object, otel-demo/NetworkPolicy/frontend-block-all-ports. Naming the failing frontend service as well would halve the score for that attempt, because the service is a symptom.