Understanding Evolution Strategies for LLM Reasoning: Broader Reasoning Coverage than GRPO

TL;DR - This paper finds that Evolution Strategies (ES) provide broader LLM reasoning coverage than GRPO, improving Pass@K while avoiding GRPO’s entropy collapse. A sequential GRPO-ES strategy combines strong Pass@1 performance with greater solution diversity.

• Verifier-projected Jensen-Shannon diversity across the ES population is theoretically and empirically associated with higher Pass@K.
• ES improves Pass@1 while achieving higher Pass@K than GRPO, which exhibits entropy collapse.
• ES gains arise from a sparse subset of large-magnitude parameter updates despite substantial whole-model drift, without necessarily causing catastrophic forgetting.
• Larger LLMs require smaller ES population sizes, informing more efficient hyperparameter design.