On Emergent Capabilities and Model Merging
Ranking
Overall
85
Content
100
Popularity
N/A
No observed public metrics; popularity remains neutral/archived.
Merged summary
TL;DR - This paper studies how model merging affects emergent capabilities across multiple testbeds and model families. It finds that shared emergent behaviors persist, absent capabilities do not become superadditive, and capabilities unique to one parent are disproportionately diluted.
- Merging two emergent-misaligned checkpoints preserves most broad misalignment across the mixing range.
- Weighted merges of single-task activation oracles never match the auditing ability of a jointly trained oracle.
- When only one parent has an emergent capability, merging weakens it faster than the associated explicitly trained capability.
- Emergent behaviors therefore compose differently—and less predictably—than trained capabilities.
Sources (1)
On Emergent Capabilities and Model Merging
Public signals
N/A
TL;DR - This paper studies how model merging affects emergent capabilities across multiple testbeds and model families. It finds that shared emergent behaviors persist, absent capabilities do not become superadditive, and capabilities unique to one parent are disproportionately diluted.
- Merging two emergent-misaligned checkpoints preserves most broad misalignment across the mixing range.
- Weighted merges of single-task activation oracles never match the auditing ability of a jointly trained oracle.
- When only one parent has an emergent capability, merging weakens it faster than the associated explicitly trained capability.
- Emergent behaviors therefore compose differently—and less predictably—than trained capabilities.