Beyond Success Rate: Cost-Aware Evaluation of Offensive and Defensive Security Agents

TL;DR - An arXiv paper arguing that security-agent benchmarks should measure economic efficiency (cost) alongside raw success, evaluating LLM agents on offensive (Cybench CTF) and defensive (Splunk BOTS v1 SOC) tasks under fixed inference/tool budgets. It matters because peak-capability-only evals overstate practical usefulness where every reasoning step and tool call costs money.

• Uses a cost-success lens, comparing models at fixed cost levels and decomposing performance by inference spend vs. tool spend rather than reporting only best-case success.
• Offensive CTF performance scales with test-time compute; scaled open-weight models can approach frontier proprietary systems while staying cost-competitive.
• Defensive SOC investigation does not scale the same way—success depends more on disciplined tool use, telemetry navigation, and selective enrichment than on raw reasoning budget.
• Advocates SOC-native, cost-aware evaluation for operational fit; results shared via an interactive website (evals.frontier.security).