‹ back to 2026-08-24

SWE Refactor Bench: Can Coding Agents Complete a Long-Horizon, Whole-Repository Stack Migration?

arXiv cs.CL LLM Agents Deyao Hong, Yizhe Chi, Wenyi Li, Xiaoqiu Wang, Mingju Gao, Kaisen Yang, Bingxiang He, Youjie Zheng, Calvin Xiao, Qinhuai Na 2026-08-24

TL;DR - SWE Refactor Bench evaluates coding agents on 20 long-horizon, whole-repository migrations using checks for both migration completeness and behavioral correctness. Frontier models succeed on only 5.4% of runs, highlighting major reliability gaps in autonomous stack migration.

The three-stage protocol combines migration audits, fixed behavioral tests, and targeted tests generated by six independent coding agents.
Only 28 of 520 runs passed all stages, and 13 of the 20 migration tasks had no accepted solution.
Of the 340 runs that completed the migration, 58% passed 99% of fixed checks, but only 26% passed every check.
Performance varied sharply by task: agents scored 31.4 on build-toolchain rewrites versus 5.6 on language rewrites; the best model scored 47.0/100.

view merged work →