Deconstructing Off-Policy Ratios: Entropy-Scaled Trust Regions for Asynchronous Reinforcement Learning
Merged summary
TL;DR - ESTR stabilizes asynchronous LLM reinforcement learning by scaling token-level off-policy trust regions according to entropy. It matches synchronous GRPO accuracy while training 2.6× faster.
- Importance-ratio behavior varies systematically with token entropy, making uniform thresholds unreliable.
- Magnitude-only correction can retain amplified low-entropy sampling noise while suppressing legitimate high-entropy exploration.
- ESTR requires neither auxiliary forward passes nor explicit policy-version detection.
- It improves train-inference consistency and outperforms existing asynchronous methods on agentic and mathematical reasoning benchmarks.
Sources (1)
Deconstructing Off-Policy Ratios: Entropy-Scaled Trust Regions for Asynchronous Reinforcement Learning
TL;DR - ESTR stabilizes asynchronous LLM reinforcement learning by scaling token-level off-policy trust regions according to entropy. It matches synchronous GRPO accuracy while training 2.6× faster.
- Importance-ratio behavior varies systematically with token entropy, making uniform thresholds unreliable.
- Magnitude-only correction can retain amplified low-entropy sampling noise while suppressing legitimate high-entropy exploration.
- ESTR requires neither auxiliary forward passes nor explicit policy-version detection.
- It improves train-inference consistency and outperforms existing asynchronous methods on agentic and mathematical reasoning benchmarks.