How Does mHC Use Its Residual Streams? Selective Routing and Near-Identity Mixing

TL;DR - An analysis of DeepSeek-V4-Flash finds that its four-stream mHC residual pathway uses selective routing and mostly near-identity mixing. The results suggest much of mHC’s theoretical flexibility is unnecessary at individual blocks, especially in later layers.

• Attention and FFN sites effectively use about two streams on average, with the dominant stream changing across depth.
• Stream representations remain directionally distinct, while cross-stream mixing is modest and concentrated in early layers.
• Replacing late-layer mixers with identity raises C4 perplexity by only 1.9% without changing the six-task average; replacing early mixers raises perplexity by 41%.
• Keeping only the three largest routing weights per token changes perplexity by at most 2.7% and the average task score by at most 0.4 points.