‹ back to 2026-08-11

SWE-Bench ProMax: Benchmarking Agents on Large-Scale Multilingual Code Refactoring

arXiv cs.CL LLM Agents Yuling Shi, Jinghan Xu, Kelin Fu, Wenhao Zeng, Shilin He, Lei Zhang, Yue Liu, Zelin Zhao, Terry Yue Zhuo, Jialun Cao, Siyu Ye, Tianyu Liu, Kai Cai, Shing-Chi Cheung, Xiaodong Gu 2026-08-10

TL;DR - SWE-Bench ProMax is an expert-curated, multilingual benchmark of 170 real-commit code-refactoring tasks designed to counter saturation and test-quality flaws in existing agent benchmarks. It matters because frontier models resolve only 41.2% of it, leaving a credible headroom signal for coding-agent progress.

Motivated by an audit finding ~60% of unsolved SWE-bench Verified instances have flawed tests (overly narrow or overly broad), plus evidence that frontier models can verbatim reproduce gold patches from training data.
Covers seven languages (Python, Java, TypeScript, Go, C, C++, Rust); each instance has issue descriptions rewritten from scratch and test suites manually reviewed to drop bad tests.
Targets large-scale, behavior-preserving changes: instances average 11.4 modified files and 261.6 lines of code, with low-complexity or single-file tasks filtered out.
Best frontier model reaches 41.2% resolve rate across two agent scaffolds; dataset is released on Hugging Face.

view merged work →