SWE Refactor Bench: Can Coding Agents Complete a Long-Horizon, Whole-Repository Stack Migration?
Ranking
Overall
89
Content
100
Popularity
65
Observed public metrics from 1 member.
Merged summary
TL;DR - SWE Refactor Bench evaluates coding agents on 20 long-horizon, whole-repository migrations using checks for both migration completeness and behavioral correctness. Frontier models succeed on only 5.4% of runs, highlighting major reliability gaps in autonomous stack migration.
- The three-stage protocol combines migration audits, fixed behavioral tests, and targeted tests generated by six independent coding agents.
- Only 28 of 520 runs passed all stages, and 13 of the 20 migration tasks had no accepted solution.
- Of the 340 runs that completed the migration, 58% passed 99% of fixed checks, but only 26% passed every check.
- Performance varied sharply by task: agents scored 31.4 on build-toolchain rewrites versus 5.6 on language rewrites; the best model scored 47.0/100.
Sources (1)
SWE Refactor Bench: Can Coding Agents Complete a Long-Horizon, Whole-Repository Stack Migration?
Public signals
Hugging Face upvotes 15 · Semantic Scholar citations 0 · Semantic Scholar influential citations 0
TL;DR - SWE Refactor Bench evaluates coding agents on 20 long-horizon, whole-repository migrations using checks for both migration completeness and behavioral correctness. Frontier models succeed on only 5.4% of runs, highlighting major reliability gaps in autonomous stack migration.
- The three-stage protocol combines migration audits, fixed behavioral tests, and targeted tests generated by six independent coding agents.
- Only 28 of 520 runs passed all stages, and 13 of the 20 migration tasks had no accepted solution.
- Of the 340 runs that completed the migration, 58% passed 99% of fixed checks, but only 26% passed every check.
- Performance varied sharply by task: agents scored 31.4 on build-toolchain rewrites versus 5.6 on language rewrites; the best model scored 47.0/100.