How Does mHC Use Its Residual Streams? Selective Routing and Near-Identity Mixing
Ranking
Overall
74
Content
90
Popularity
37
Observed public metrics from 1 member.
Merged summary
TL;DR - An analysis of DeepSeek-V4-Flash finds that its four-stream mHC residual pathway uses selective routing and mostly near-identity mixing. The results suggest much of mHC’s theoretical flexibility is unnecessary at individual blocks, especially in later layers.
- Attention and FFN sites effectively use about two streams on average, with the dominant stream changing across depth.
- Stream representations remain directionally distinct, while cross-stream mixing is modest and concentrated in early layers.
- Replacing late-layer mixers with identity raises C4 perplexity by only 1.9% without changing the six-task average; replacing early mixers raises perplexity by 41%.
- Keeping only the three largest routing weights per token changes perplexity by at most 2.7% and the average task score by at most 0.4 points.
Sources (1)
How Does mHC Use Its Residual Streams? Selective Routing and Near-Identity Mixing
Public signals
Semantic Scholar citations 0 · Semantic Scholar influential citations 0
TL;DR - An analysis of DeepSeek-V4-Flash finds that its four-stream mHC residual pathway uses selective routing and mostly near-identity mixing. The results suggest much of mHC’s theoretical flexibility is unnecessary at individual blocks, especially in later layers.
- Attention and FFN sites effectively use about two streams on average, with the dominant stream changing across depth.
- Stream representations remain directionally distinct, while cross-stream mixing is modest and concentrated in early layers.
- Replacing late-layer mixers with identity raises C4 perplexity by only 1.9% without changing the six-task average; replacing early mixers raises perplexity by 41%.
- Keeping only the three largest routing weights per token changes perplexity by at most 2.7% and the average task score by at most 0.4 points.