HarnessOpt-Bench: Evaluating LLMs at Harness Optimization

TL;DR - HarnessOpt-Bench is a benchmark measuring how well frontier LLMs can automatically optimize an agent's "harness" (prompts, tools, control flow, memory, orchestration code) under expensive, stochastic evaluation. It matters because agentic system performance depends heavily on the scaffolding around the model, and there was no common protocol to measure this self-improvement capability.

• Setup: an optimizer LLM plus coding harness receives a seed harness, graded evaluation feedback, and a fixed target-evaluation budget; it edits the harness and nominates one final candidate.
• Scoring: normalized gain over the seed on a held-out test partition kept inaccessible during search, with a trusted execution environment enforcing the evaluation boundary, metering target-agent resource use, and preserving candidate versions for audit.
• Scale: 5 frontier LLMs evaluated as optimizers under both a shared coding harness and their native harnesses across 4 downstream tasks, totaling 111 scored runs.
• Findings: optimizer models separate more than the coding harnesses they act through, native harnesses are not consistently better, and gains vary substantially by task and seed regime — establishing harness optimization as discriminative with large headroom.