ASIL: Replacing Screenshot-and-Click with Structured State and Semantic Actions

TL;DR - ASIL replaces screenshot-based GUI control with structured JSON state and executable semantic actions, enabling software agents to complete tasks with far fewer actions and substantially higher success rates. The interface also provides an effective training environment for improving smaller agent models through supervised fine-tuning and reinforcement learning.

• ASIL was implemented across 15 applications and evaluated on 300 single-application and 80 multi-application tasks.
• Closed models achieved over 80% success while using fewer than five actions per task.
• Screenshot-and-click baselines reached only 6.6% and 26.6% strict success under a 50-step budget, or 15.0% and 53.3% on an easier task band.
• Fine-tuning raised Qwen3.5-2B from 58.0% to 72.1% and Qwen3.5-9B from 66.6% to 80.4%; on-policy RL further improved them to 74.4% and 82.2%.