在数学上把稀疏注意力做对！腾讯Hy开源HiLS-Attention: 计算更少效果更好, 外推512倍

TL;DR - Tencent Hunyuan open-sourced HiLS-Attention, a hierarchical sparse-attention method designed for efficient long-context modeling. It reportedly preserves full-attention quality while extrapolating from 8K training contexts to 4M tokens and substantially accelerating inference.

• Uses Taylor-based chunk scoring with an entropy bias to better approximate attention importance than mean- or max-logit methods.
• A hierarchical softmax makes chunk summaries trainable end to end through the language-modeling loss.
• Across 345M–7B models, short-context perplexity reportedly remains close to full attention; 8K-trained models retain over 90% needle-retrieval accuracy at 4M tokens.
• At 512K context, reported speedups are 13.5× for prefill and 15.7× for single-step decoding.