把思考折叠进序列：WeLM 617B MoE的隐式Scaling路径

TL;DR - WeChat’s WeLM team scaled Hidden Decoding to a 617B-parameter MoE model, enabling parallel latent computation between tokens without lengthening visible outputs. It reportedly beat matched autoregressive baselines on all nine evaluations after limited continued pretraining.

• Hidden Decoding expands each token into parallel streams processed by shared Transformer weights, with prediction loss applied only to the final stream.
• Stream-Factorized Attention keeps added attention cost near-linear rather than quadratic in the number of streams.
• At four streams, batch time was 4.4× baseline for the 617B model, versus 16× for dense attention.
• The 617B model improved its evaluation average by about 1.03 percentage points using continued pretraining equal to 5.3% of full training volume.