Deconstructing Off-Policy Ratios: Entropy-Scaled Trust Regions for Asynchronous Reinforcement Learning
Ranking
Overall
84
Content
90
Popularity
71
Observed public metrics from 1 member.
Merged summary
TL;DR - ESTR stabilizes asynchronous LLM reinforcement learning by scaling token-level off-policy trust regions according to entropy. It matches synchronous GRPO accuracy while training 2.6Ă— faster.
- Importance-ratio behavior varies systematically with token entropy, making uniform thresholds unreliable.
- Magnitude-only correction can retain amplified low-entropy sampling noise while suppressing legitimate high-entropy exploration.
- ESTR requires neither auxiliary forward passes nor explicit policy-version detection.
- It improves train-inference consistency and outperforms existing asynchronous methods on agentic and mathematical reasoning benchmarks.
Sources (1)
Deconstructing Off-Policy Ratios: Entropy-Scaled Trust Regions for Asynchronous Reinforcement Learning
Public signals
Semantic Scholar citations 1 · Semantic Scholar influential citations 0
TL;DR - ESTR stabilizes asynchronous LLM reinforcement learning by scaling token-level off-policy trust regions according to entropy. It matches synchronous GRPO accuracy while training 2.6Ă— faster.
- Importance-ratio behavior varies systematically with token entropy, making uniform thresholds unreliable.
- Magnitude-only correction can retain amplified low-entropy sampling noise while suppressing legitimate high-entropy exploration.
- ESTR requires neither auxiliary forward passes nor explicit policy-version detection.
- It improves train-inference consistency and outperforms existing asynchronous methods on agentic and mathematical reasoning benchmarks.