JarvisGUI: Towards Cross-Device GUI Agents with Dynamic Task Composition

TL;DR - JarvisGUI is a dynamic benchmark for evaluating GUI agents on workflows spanning Android, Windows, and Ubuntu. It exposes major gaps in agents’ ability to transfer state, reason across platforms, and manage long-horizon dependencies.

• Models tasks as typed input-output transformations, enabling automatic composition of multi-step cross-device workflows.
• Evaluates coordinated agent interactions in virtual environments across three operating systems.
• Addresses limitations of existing benchmarks that focus on static, single-device tasks.
• Finds that state-of-the-art open-source GUI agents struggle with cross-device coordination requirements.