‹ back to 2026-09-09

ExecCritic: Learn to Test, Test to Improve for Coding Agents

arXiv cs.AI LLM Agents Leitian Tao, Baolin Peng, Haorui Wang, Hang Wang, Hao Cheng, Wenlin Yao, Qianhui Wu, Tao Ge, Sharon Li, Jianfeng Gao 2026-09-08

TL;DR - ExecCritic trains separate test-writing and code-repair agents so execution feedback reflects issue requirements rather than reinforcing shared mistakes. On SWE-bench Verified, this role-specific approach raises resolution from a 61.2% no-test baseline to 72.6%.

A Test agent generates repository-native tests, while a fail-closed harness validates and freezes them before repair begins.
A separately trained Repair agent uses test execution feedback to revise source code but cannot modify the tests.
Weak generated tests hurt performance: base Qwen tests reduce resolution to 57.3%, while GPT-5.6-sol tests increase it to 65.3%.
Post-training improves the Qwen Test agent’s Base-to-Gold success from 22.2% to 62.2%; combining both post-trained Qwen agents delivers an 11.4-point gain without stronger-model or Oracle feedback at evaluation time.

view merged work →