Difficulty-Adaptive Tree-Structured Policy Optimization for Expanding Reasoning Coverage in RLVR

TL;DR - DATPO is an RLVR method that uses difficulty-adaptive, sentence-entropy-guided tree rollouts to broaden language models’ reasoning coverage. It improves pass@k on mathematical reasoning benchmarks, enabling stronger test-time scaling.

• Difficulty-adaptive rollouts help expand pass@k rather than serving only as an efficiency optimization.
• Tree-based rollouts discover correct answers more effectively than parallel sampling.
• Sentence-level entropy guides branching toward semantically diverse reasoning paths, avoiding localized token-level variations.
• A sibling-diversity advantage explicitly rewards diversity among branches during policy optimization.