ORCA-bench: How Ready Are Language Model Agents for Oncall?

TL;DR - ORCA-bench evaluates coding agents on production-like oncall root cause analysis using real telemetry interfaces and source code. Frontier agents achieve only 25.3% accuracy on medium tasks and 10.0% on hard tasks, indicating they are not yet reliable for production operations.

• Includes 1,079 tasks over a six-day, 50 GB OpenTelemetry-instrumented microservice testbed.
• Tasks vary report specificity, detection delay, and combinations of concurrent faults.
• Ground-truth symptoms were expert-reviewed, with strong human agreement on judge rescoring (weighted Cohen’s κ = 0.90).
• Removing source-code access degraded every metric; the weakest model hallucinated implausible causes in 40% of reports.