SWE-Bench ProMax: Benchmarking Agents on Large-Scale Multilingual Code Refactoring 🔗 2 sources
TL;DR — SWE-Bench ProMax is an expert-curated, multilingual benchmark of 170 real-commit code-refactoring tasks that extends the SWE-Bench lineage beyond Python-centric, single-file bug fixing toward large-scale, behavior-preserving software maintenance. It matters because frontier models resolve only 41.2% of it, restoring credible headroom after saturation and test-quality problems in existing agent benchmarks.
- Motivation: An audit found ~60% of unsolved SWE-bench Verified instances have flawed tests (overly narrow or overly broad), and frontier models can reproduce gold patches verbatim from training data — so ProMax rewrites every issue description from scratch and manually reviews test suites to drop bad tests.
- Multilingual scope: Seven languages (Python, Java, TypeScript, Go, C, C++, Rust), moving agent evaluation past SWE-Bench's original Python focus.
- Large-scale tasks: Instances average 11.4 modified files and 261.6 lines changed, with low-complexity and single-file tasks filtered out.
- Refactoring, not bug-fixing: The focus on behavior-preserving edits stresses cross-file reasoning and API/call-site consistency, capabilities poorly measured by existing benchmarks.
- Results and release: The best frontier model reaches a 41.2% resolve rate across two agent scaffolds; the dataset is released on Hugging Face.
Emphasis differs: the arXiv entry supplies the concrete methodology, dataset statistics, and results, while the @_akhaliq post is only a title-plus-link announcement whose framing (difficulty axes, refactoring challenges) is inferred from the title alone.