An Empirical Study of Harness Design for Coding Agents

TL;DR - This empirical study isolates how planning, action space, and context management affect autonomous coding agents across four models and two software-engineering benchmarks. It shows that effective harness design depends on model capability and context budget rather than a universally optimal configuration.

• Context management matters most under tight context budgets, primarily by preventing context-overflow failures and enabling longer trajectories.
• Rule-based elision followed by LLM summarization provides the best overall efficiency; recoverable elisions add complexity without improving accuracy.
• Planning improves accuracy for weaker models but mainly reduces costs for stronger models, with little accuracy impact.
• Predefined tools help models with weaker bash skills, while bash-capable models achieve similar effectiveness at substantially lower cost with a bash-only interface.