MoE-ViE: Mixture of Experts Vision Encoder for Efficient Image and Video Understanding

TL;DR - MoE-ViE efficiently scales CLIP-style vision encoders using fine-grained mixture-of-experts designs, specialized load balancing, and an optimized inference kernel. Its largest model matches a state-of-the-art encoder 1.7× larger while using 76% of its latency.

• Fine-grained MoE topologies consistently outperform dense and standard MoE vision encoders across model sizes.
• An auxiliary-loss-free balancing method improves expert utilization, while a specialized MoE kernel reduces inference overhead.
• Frame-level distillation and selective freezing add video capabilities while preserving learned image knowledge.
• When aligned with an LLM, MoE-ViE leads the compared image and video benchmarks, including against encoders with up to 5× more activated parameters.