Plover: Steering GUI Agents through Plan-Centric Interaction

TL;DR - Plover is a plan-centric, vision-based GUI automation system that externalizes an agent's plans and replanning as inspectable, editable artifacts, letting users supervise and correct execution mid-task. It matters because it turns opaque autonomous GUI agents into transparent, human-steerable workflows.

• Uses a planner–executor architecture that surfaces task plans and replanning as persistent, revisable artifacts rather than hidden internal state.
• Supports localized human interventions: editable plans, natural-language guidance, and screenshot-grounded corrections, while preserving prior progress during repair.
• Interaction design was informed by a formative study with six participants.
• Evaluation via benchmark failure-case repair and scenario-based analyses found many autonomous GUI-agent failures are structurally repairable when plans stay visible and interventions are localized.