SWE-Bench ProMax: Benchmarking Agents on Large-Scale Multilingual Code Refactoring
Ranking
Observed public metrics from 1 member.
Merged summary
TL;DR — SWE-Bench ProMax is an expert-curated, multilingual benchmark of 170 real-commit code-refactoring tasks that extends the SWE-Bench lineage beyond Python-centric, single-file bug fixing toward large-scale, behavior-preserving software maintenance. It matters because frontier models resolve only 41.2% of it, restoring credible headroom after saturation and test-quality problems in existing agent benchmarks.
- Motivation: An audit found ~60% of unsolved SWE-bench Verified instances have flawed tests (overly narrow or overly broad), and frontier models can reproduce gold patches verbatim from training data — so ProMax rewrites every issue description from scratch and manually reviews test suites to drop bad tests.
- Multilingual scope: Seven languages (Python, Java, TypeScript, Go, C, C++, Rust), moving agent evaluation past SWE-Bench's original Python focus.
- Large-scale tasks: Instances average 11.4 modified files and 261.6 lines changed, with low-complexity and single-file tasks filtered out.
- Refactoring, not bug-fixing: The focus on behavior-preserving edits stresses cross-file reasoning and API/call-site consistency, capabilities poorly measured by existing benchmarks.
- Results and release: The best frontier model reaches a 41.2% resolve rate across two agent scaffolds; the dataset is released on Hugging Face.
Emphasis differs: the arXiv entry supplies the concrete methodology, dataset statistics, and results, while the @_akhaliq post is only a title-plus-link announcement whose framing (difficulty axes, refactoring challenges) is inferred from the title alone.
Sources (2)
SWE-Bench ProMax: Benchmarking Agents on Large-Scale Multilingual Code Refactoring
TL;DR - SWE-Bench ProMax is an expert-curated, multilingual benchmark of 170 real-commit code-refactoring tasks designed to counter saturation and test-quality flaws in existing agent benchmarks. It matters because frontier models resolve only 41.2% of it, leaving a credible headroom signal for coding-agent progress.
- Motivated by an audit finding ~60% of unsolved SWE-bench Verified instances have flawed tests (overly narrow or overly broad), plus evidence that frontier models can verbatim reproduce gold patches from training data.
- Covers seven languages (Python, Java, TypeScript, Go, C, C++, Rust); each instance has issue descriptions rewritten from scratch and test suites manually reviewed to drop bad tests.
- Targets large-scale, behavior-preserving changes: instances average 11.4 modified files and 261.6 lines of code, with low-complexity or single-file tasks filtered out.
- Best frontier model reaches 41.2% resolve rate across two agent scaffolds; dataset is released on Hugging Face.
SWE-Bench ProMax Benchmarking Agents on Large-Scale Multilingual Code Refactoring paper…
TL;DR - A shared announcement of "SWE-Bench ProMax," a paper introducing a benchmark that evaluates coding agents on large-scale, multilingual code refactoring tasks. It matters because it pushes agent evaluation beyond single-repo, Python-centric bug fixing toward broader, more realistic software maintenance work.
- Positions itself as an extension of the SWE-Bench lineage, which evaluates LLM agents on resolving real repository issues; "ProMax" signals a harder/expanded tier.
- Two stated axes of difficulty: large-scale changes (edits spanning many files/modules rather than small patches) and multilingual coverage (beyond SWE-Bench's original Python focus).
- Targets refactoring rather than bug-fixing, which stresses cross-file reasoning, API/call-site consistency, and behavior-preserving edits — capabilities poorly measured by existing agent benchmarks.
- Content is thin: the item is only a paper title plus a truncated Hugging Face Papers link (arXiv ID 2608.0…), so no methodology, task counts, or model results are available; the above is inferred from the title alone.