🛰️ Daily AI Frontier
‹ back to 2026-09-19

Don't Mask the Environment: Observation Supervision Changes How Agents Explore Under RL

Research LLM Agents

Ranking

Overall 89
Content 100
Popularity 64

Observed public metrics from 1 member.

Merged summary

TL;DR - ActObs improves agent initialization for reinforcement learning by supervising both action and environment-observation tokens during fine-tuning. This preserves consequence prediction and encourages broader exploration without additional data, parameters, tokens, or forward passes.

  • ActObs and action-only training perform similarly after SFT but diverge after GRPO reinforcement learning.
  • On Terminal-Bench 2.0, ActObs improves pass@k across all tested sampling budgets for Qwen3-4B; with Qwen3-8B, it gains 3.4 percentage points at pass@16 while sacrificing some pass@1 reliability.
  • The method generalizes to unseen cross-domain code editing, improving aider-polyglot pass@1 by 4.2 percentage points with Qwen3-4B.
  • ActObs retains more policy entropy and requires less policy movement during RL, while action-only SFT degrades the base model’s ability to predict environmental consequences.

Sources (1)

Don't Mask the Environment: Observation Supervision Changes How Agents Explore Under RL

arXiv cs.LG Juzheng Zhang, Disha Makhija, Manoj Ghuhan Arivazhagan, Vinayshekhar Bannihatti Kumar, Rashmi Gangadharaiah 2026-09-17 arXiv:2609.20715
Public signals Hugging Face upvotes 42
Providers: Hugging Face · Upvotes 42 OpenAlex · N/A Publisher · N/A Semantic Scholar · N/A X · N/A Fetched 2026-09-25 14:18:05.622369 UTC

TL;DR - ActObs improves agent initialization for reinforcement learning by supervising both action and environment-observation tokens during fine-tuning. This preserves consequence prediction and encourages broader exploration without additional data, parameters, tokens, or forward passes.

  • ActObs and action-only training perform similarly after SFT but diverge after GRPO reinforcement learning.
  • On Terminal-Bench 2.0, ActObs improves pass@k across all tested sampling budgets for Qwen3-4B; with Qwen3-8B, it gains 3.4 percentage points at pass@16 while sacrificing some pass@1 reliability.
  • The method generalizes to unseen cross-domain code editing, improving aider-polyglot pass@1 by 4.2 percentage points with Qwen3-4B.
  • ActObs retains more policy entropy and requires less policy movement during RL, while action-only SFT degrades the base model’s ability to predict environmental consequences.
item →