Don't Mask the Environment: Observation Supervision Changes How Agents Explore Under RL

TL;DR - ActObs improves agent initialization for reinforcement learning by supervising both action and environment-observation tokens during fine-tuning. This preserves consequence prediction and encourages broader exploration without additional data, parameters, tokens, or forward passes.

• ActObs and action-only training perform similarly after SFT but diverge after GRPO reinforcement learning.
• On Terminal-Bench 2.0, ActObs improves pass@k across all tested sampling budgets for Qwen3-4B; with Qwen3-8B, it gains 3.4 percentage points at pass@16 while sacrificing some pass@1 reliability.
• The method generalizes to unseen cross-domain code editing, improving aider-polyglot pass@1 by 4.2 percentage points with Qwen3-4B.
• ActObs retains more policy entropy and requires less policy movement during RL, while action-only SFT degrades the base model’s ability to predict environmental consequences.