万字长文详解 DeepSeek、Llama、GLM 底层架构设计

TL;DR - A technical explainer compares the architectural evolution from Llama to DeepSeek, GLM, and Qwen, framing recent advances as efficiency improvements rather than wholesale Transformer redesigns. It shows how modern models increase capacity and context length while controlling inference memory and computation.

• MLA compresses key-value representations to reduce KV-cache memory, while decoupled RoPE preserves positional information.
• Sparse attention mechanisms such as DSA lower the computational cost of long sequences.
• Mixture-of-Experts models expand parameter capacity without activating every parameter for each token, though they must balance traffic across experts.
• RMSNorm, pre-normalization, GQA, QK-Norm, RoPE, and FlashAttention improve training stability or attention efficiency across modern LLMs.