SWE-bench Science: Can Coding Agents Resolve Engineering Tasks in Science?

TL;DR - SWE-bench Science is a 119-task benchmark evaluating coding agents on repository-level engineering work across 20 scientific domains. Its strongest tested agent scores below 50% pass@1, exposing persistent difficulties with scientific reasoning, exploration, complete integration, and generalization.

• The benchmark draws tasks from 98 GitHub repositories and covers issue-driven, expert-exploratory, and engineering-integration scenarios.
• Claude Code with Opus-5 (max), the best-performing evaluated agent, achieves less than 50% pass@1.
• Common failures include missing scientific abstractions, superficial or misguided repairs, incomplete system-level coverage, and poor generalization beyond observed cases.
• Ablations show that well-grounded scientific guidance can improve average performance and token efficiency, while misaligned guidance can anchor agents without improving exact repair success.