Understanding Evolution Strategies for LLM Reasoning: Broader Reasoning Coverage than GRPO
Ranking
Overall
82
Content
90
Popularity
63
Observed public metrics from 1 member.
Merged summary
TL;DR - This paper finds that Evolution Strategies (ES) provide broader LLM reasoning coverage than GRPO, improving Pass@K while avoiding GRPO’s entropy collapse. A sequential GRPO-ES strategy combines strong Pass@1 performance with greater solution diversity.
- Verifier-projected Jensen-Shannon diversity across the ES population is theoretically and empirically associated with higher Pass@K.
- ES improves Pass@1 while achieving higher Pass@K than GRPO, which exhibits entropy collapse.
- ES gains arise from a sparse subset of large-magnitude parameter updates despite substantial whole-model drift, without necessarily causing catastrophic forgetting.
- Larger LLMs require smaller ES population sizes, informing more efficient hyperparameter design.
Sources (1)
Understanding Evolution Strategies for LLM Reasoning: Broader Reasoning Coverage than GRPO
Public signals
Hugging Face upvotes 21
TL;DR - This paper finds that Evolution Strategies (ES) provide broader LLM reasoning coverage than GRPO, improving Pass@K while avoiding GRPO’s entropy collapse. A sequential GRPO-ES strategy combines strong Pass@1 performance with greater solution diversity.
- Verifier-projected Jensen-Shannon diversity across the ES population is theoretically and empirically associated with higher Pass@K.
- ES improves Pass@1 while achieving higher Pass@K than GRPO, which exhibits entropy collapse.
- ES gains arise from a sparse subset of large-magnitude parameter updates despite substantial whole-model drift, without necessarily causing catastrophic forgetting.
- Larger LLMs require smaller ES population sizes, informing more efficient hyperparameter design.