ORCA-bench: How Ready Are Language Model Agents for Oncall?
Ranking
Overall
81
Content
100
Popularity
37
Observed public metrics from 1 member.
Merged summary
TL;DR - ORCA-bench evaluates coding agents on production-like oncall root cause analysis using real telemetry interfaces and source code. Frontier agents achieve only 25.3% accuracy on medium tasks and 10.0% on hard tasks, indicating they are not yet reliable for production operations.
- Includes 1,079 tasks over a six-day, 50 GB OpenTelemetry-instrumented microservice testbed.
- Tasks vary report specificity, detection delay, and combinations of concurrent faults.
- Ground-truth symptoms were expert-reviewed, with strong human agreement on judge rescoring (weighted Cohen’s κ = 0.90).
- Removing source-code access degraded every metric; the weakest model hallucinated implausible causes in 40% of reports.
Sources (1)
ORCA-bench: How Ready Are Language Model Agents for Oncall?
Public signals
Semantic Scholar citations 0 · Semantic Scholar influential citations 0
TL;DR - ORCA-bench evaluates coding agents on production-like oncall root cause analysis using real telemetry interfaces and source code. Frontier agents achieve only 25.3% accuracy on medium tasks and 10.0% on hard tasks, indicating they are not yet reliable for production operations.
- Includes 1,079 tasks over a six-day, 50 GB OpenTelemetry-instrumented microservice testbed.
- Tasks vary report specificity, detection delay, and combinations of concurrent faults.
- Ground-truth symptoms were expert-reviewed, with strong human agreement on judge rescoring (weighted Cohen’s κ = 0.90).
- Removing source-code access degraded every metric; the weakest model hallucinated implausible causes in 40% of reports.