‹ back to 2026-09-04

SWE-Gate: Passing Functional Tests Is Not Enough for Software Engineering Agents

arXiv cs.SE LLM Agents Xin He, Yanlin Wang, Mingwei Liu, Jiachi Chen, Hongyu Zhang, Guanbin Li 2026-09-03

TL;DR - SWE-Gate is a repository-level benchmark that evaluates whether coding agents satisfy review-derived constraints in addition to passing functional tests. Its results show that functional-only benchmarks substantially overestimate agents’ ability to produce acceptable real-world patches.

Contains 303 repair instances from 75 open-source Python repositories.
Separates functional tests from tests for constraints derived from pull request review comments.
Of 644 agent-generated repairs that passed functional tests, 221 failed the review constraints.
Includes non-compliant and gold patches, with code, data, and experimental results publicly available.

view merged work →