114B参数、6B激活，Sand.ai刚刚开源全球首个千亿MoE视频生成模型

TL;DR - Sand.ai open-sourced MAGI-2-preview, a 114B-parameter MoE model for joint video and audio generation that activates about 6B parameters per forward pass. It provides researchers and enterprises with an unusually large open model for studying, fine-tuning, and privately deploying video MoE systems.

• A single-stream Transformer jointly models text, video, and audio, enabling direct cross-modal interaction at every self-attention layer.
• Multi-Head Latent MoE splits hidden states into 12 independently routed heads; each layer contains 3,072 expert units and activates 72 per token.
• Sand.ai developed custom kernels and head-parallel execution to reduce routing overhead and keep communication independent of the number of activated experts.
• The article reports sixth place on the AA video-generation leaderboard and an estimated cost of roughly ¥0.50 for a 10-second 1080p clip on eight H100 GPUs.