🛰️ Daily AI Frontier
‹ back to 2026-09-09

ExecCritic: Learn to Test, Test to Improve for Coding Agents

Research LLM Agents

Ranking

Overall 81
Content 100
Popularity 37

Observed public metrics from 1 member.

Merged summary

TL;DR - ExecCritic trains separate test-writing and code-repair agents so execution feedback reflects issue requirements rather than reinforcing shared mistakes. On SWE-bench Verified, this role-specific approach raises resolution from a 61.2% no-test baseline to 72.6%.

  • A Test agent generates repository-native tests, while a fail-closed harness validates and freezes them before repair begins.
  • A separately trained Repair agent uses test execution feedback to revise source code but cannot modify the tests.
  • Weak generated tests hurt performance: base Qwen tests reduce resolution to 57.3%, while GPT-5.6-sol tests increase it to 65.3%.
  • Post-training improves the Qwen Test agent’s Base-to-Gold success from 22.2% to 62.2%; combining both post-trained Qwen agents delivers an 11.4-point gain without stronger-model or Oracle feedback at evaluation time.

Sources (1)

ExecCritic: Learn to Test, Test to Improve for Coding Agents

arXiv cs.AI Leitian Tao, Baolin Peng, Haorui Wang, Hang Wang, Hao Cheng, Wenlin Yao, Qianhui Wu, Tao Ge, Sharon Li, Jianfeng Gao 2026-09-08 arXiv:2609.09133
Public signals Semantic Scholar citations 0 · Semantic Scholar influential citations 0
Providers: Hugging Face · N/A OpenAlex · N/A Publisher · N/A Semantic Scholar · Citations 0 · Influential citations 0 X · N/A Fetched 2026-09-22 14:24:35.107562 UTC

TL;DR - ExecCritic trains separate test-writing and code-repair agents so execution feedback reflects issue requirements rather than reinforcing shared mistakes. On SWE-bench Verified, this role-specific approach raises resolution from a 61.2% no-test baseline to 72.6%.

  • A Test agent generates repository-native tests, while a fail-closed harness validates and freezes them before repair begins.
  • A separately trained Repair agent uses test execution feedback to revise source code but cannot modify the tests.
  • Weak generated tests hurt performance: base Qwen tests reduce resolution to 57.3%, while GPT-5.6-sol tests increase it to 65.3%.
  • Post-training improves the Qwen Test agent’s Base-to-Gold success from 22.2% to 62.2%; combining both post-trained Qwen agents delivers an 11.4-point gain without stronger-model or Oracle feedback at evaluation time.
item →