Task-CoEvolve: Efficient Harness Optimization via Adaptive Validation Task Selection

TL;DR - Task-CoEvolve reduces the cost of optimizing LLM agent harnesses by adaptively evaluating the validation tasks that best distinguish candidate harnesses. It matches full-validation-set search performance while using 80% fewer evaluations.

• Samples tasks using outcome variance, emphasizing examples near the agent’s evolving capability frontier.
• Estimates full-set performance from partial evaluations by correcting for each task’s sampling probability.
• Enables consistent candidate comparisons even when different validation subsets are used across iterations.
• Outperforms fixed-subset baselines on online text classification and Terminal-Bench 2.1.