The Dark Room in the Reward Channel: Dense Prediction Rewards Collapse GRPO-Trained LLM Agents -- and What Actually Works

TL;DR - Dense prediction rewards can collapse GRPO-trained LLM agents because groupwise standard-deviation normalization amplifies reward variance in all-fail groups. Delivering the same signal through an auxiliary loss instead improved ALFWorld performance by roughly 20 points.

• Across Qwen3-1.7B/4B/8B, prediction accuracy reached 1.0 while task success fell to 0% and episodes hit the maximum horizon.
• Removing only GRPO’s standard-deviation normalization restored performance to baseline parity.
• The authors propose that dense signals are safe when their within-group variance declines as the model masters them.
• Results are currently single-seed; replication and group-size controls are preregistered and ongoing.