OSWorld-Pro: Process-based Evaluation for Computer Use Agents

TL;DR - OSWorld-Pro is a process-based benchmark for evaluating computer-use agents across intermediate subgoals rather than only final task outcomes. It exposes specific failure modes that end-state evaluation can obscure, helping target improvements in agent reliability and efficiency.

• Includes 300+ tasks, 2,800+ subgoals, and over 67,000 human annotations.
• Uses human-aligned LLM judges to assess progress through sequentially dependent subgoals.
• Claude Opus 5, the top reported performer, scored 75.7% on OSWorld-Pro versus 83.4% on OSWorld.
• Process-level analysis identifies failures such as subgoal-irrelevant actions, keyboard-input errors, and imprecise GUI clicks.