把思考折叠进序列：WeLM 617B MoE的隐式Scaling路径
Merged summary
TL;DR - WeChat’s WeLM team scaled Hidden Decoding to a 617B-parameter MoE model, enabling parallel latent computation between tokens without lengthening visible outputs. It reportedly beat matched autoregressive baselines on all nine evaluations after limited continued pretraining.
- Hidden Decoding expands each token into parallel streams processed by shared Transformer weights, with prediction loss applied only to the final stream.
- Stream-Factorized Attention keeps added attention cost near-linear rather than quadratic in the number of streams.
- At four streams, batch time was 4.4× baseline for the 617B model, versus 16× for dense attention.
- The 617B model improved its evaluation average by about 1.03 percentage points using continued pretraining equal to 5.3% of full training volume.
Sources (1)
把思考折叠进序列：WeLM 617B MoE的隐式Scaling路径
TL;DR - WeChat’s WeLM team scaled Hidden Decoding to a 617B-parameter MoE model, enabling parallel latent computation between tokens without lengthening visible outputs. It reportedly beat matched autoregressive baselines on all nine evaluations after limited continued pretraining.
- Hidden Decoding expands each token into parallel streams processed by shared Transformer weights, with prediction loss applied only to the final stream.
- Stream-Factorized Attention keeps added attention cost near-linear rather than quadratic in the number of streams.
- At four streams, batch time was 4.4× baseline for the 617B model, versus 16× for dense attention.
- The 617B model improved its evaluation average by about 1.03 percentage points using continued pretraining equal to 5.3% of full training volume.