One Frozen Simulator Is Not Enough: Simulator Collapse in Multi-Agent RL

TL;DR - This paper identifies “simulator collapse,” where agent policies overfit to a single mode-collapsed LLM user simulator. Diversifying simulator behavior improves generalization to unseen simulators and real users.

• Verbalized Sampling broadens simulator responses at inference time, improving held-out success by up to 9%.
• Co-Training jointly trains policies against multiple simulators, increasing gains to 14%.
• Both methods preserve policy diversity across three multi-turn benchmarks.
• The authors release SCOPE, an open-source framework for population co-training in multi-agent RL.