LoopArena: Benchmarking Models as Runtime Controllers for Loop Engineering

TL;DR - LoopArena benchmarks language models acting as runtime Controllers that guide a fixed coding agent through long-running development tasks. Its best full-task strict success rate is only 24.69%, highlighting substantial room to improve agentic loop control.

• Separates the Controller’s guidance quality from the Worker coding agent’s execution ability.
• Evaluates control at three levels: execution-validated next-step selection, repeated control over task slices, and complete end-to-end tasks.
• Controllers reduce estimated inference costs by an average of 64.4% in paired comparisons.
• The lower-cost Type II evaluation closely matches the main controller ranking, with Spearman’s ρ of 0.9747.