ESTR entropy-scaled trust region stabilizes asynchronous LLM reinforcement learning

An arXiv preprint proposes Entropy-Scaled Trust Region, or ESTR, to stabilize asynchronous reinforcement learning used in large language model post-training when rollout generation overlaps policy optimization. The authors argue that stale off-policy data can destabilize optimization and cause policy collapse. Existing methods often keep or drop tokens using only the magnitude of importance ratios with one threshold across positions. They report that the natural scale of the importance ratio varies with token entropy: at low entropy, train-inference discrepancy is amplified into sampling noise, while at high entropy in-flight weight updates produce legitimate exploratory deviations. Magnitude-only correction can admit that noise while masking exploration from in-flight updates. ESTR scales each token's off-policy deviation by its local entropy and needs no auxiliary forward passes or explicit version-switch detection. Across long-horizon agentic tasks and mathematical reasoning benchmarks, the paper says ESTR outperformed existing asynchronous methods and achieved the best train-inference consistency. Reported scores include 37.34 avg@1 on BrowseComp-Plus and 95.69 on multi-turn GSM8K, matching synchronous GRPO with a 2.6x speedup.