TPAMI | 北大 & 清华 & 复旦 提出 SparseVLM+：修正注意力偏置，让「文本引导的视觉稀疏化」更精准
Ranking
No observed public metrics; popularity remains neutral/archived.
Merged summary
TL;DR - SparseVLM+ (Peking U., Tsinghua, Fudan; TPAMI journal extension of ICML 2025's SparseVLM) is a training-free, plug-and-play method for pruning visual tokens in vision-language models that first corrects the attention signal used to rank tokens. It matters because attention-based pruning is the dominant efficiency trick for VLMs, and this work shows the attention scores themselves are systematically biased.
- Two diagnosed biases: Attention Gravity — RoPE-style position encoding makes text tokens over-attend to physically nearby visual tokens rather than semantically relevant ones (strongest in shallow layers); and Attention Sink — semantically irrelevant visual tokens absorb disproportionate attention regardless of the query.
- Fixes: gravity correction estimates the pure positional prior by running uniformly-initialized queries/keys, then divides it out of the real attention map; Priority Heads Selection keeps only the Top-K attention heads with the highest weighted cross-modal scores, discarding noisy heads.
- Video extension: Temporal Balanced Sparsification combines corrected text-relevance with a temporal diversity score (nearest-neighbor Euclidean distance to already-selected tokens), keeping Top-K tokens per frame.
- Reported results: LLaVA-1.5-7B retains 99.6% performance at 66.7% token compression (192 tokens), +1.5% over SparseVLM and +1.6% over VisPruner; 96.9% retained at 80.2% pruning; Qwen2.5-VL gains 0.5 on MathVista at ~60.5% reduction; LLaVA-OneVision-7B reaches 96.2% with only 15% of tokens vs. 94.6% for FrameFusion.
Sources (1)
TPAMI | 北大 & 清华 & 复旦 提出 SparseVLM+：修正注意力偏置，让「文本引导的视觉稀疏化」更精准
TL;DR - SparseVLM+ (Peking U., Tsinghua, Fudan; TPAMI journal extension of ICML 2025's SparseVLM) is a training-free, plug-and-play method for pruning visual tokens in vision-language models that first corrects the attention signal used to rank tokens. It matters because attention-based pruning is the dominant efficiency trick for VLMs, and this work shows the attention scores themselves are systematically biased.
- Two diagnosed biases: Attention Gravity — RoPE-style position encoding makes text tokens over-attend to physically nearby visual tokens rather than semantically relevant ones (strongest in shallow layers); and Attention Sink — semantically irrelevant visual tokens absorb disproportionate attention regardless of the query.
- Fixes: gravity correction estimates the pure positional prior by running uniformly-initialized queries/keys, then divides it out of the real attention map; Priority Heads Selection keeps only the Top-K attention heads with the highest weighted cross-modal scores, discarding noisy heads.
- Video extension: Temporal Balanced Sparsification combines corrected text-relevance with a temporal diversity score (nearest-neighbor Euclidean distance to already-selected tokens), keeping Top-K tokens per frame.
- Reported results: LLaVA-1.5-7B retains 99.6% performance at 66.7% token compression (192 tokens), +1.5% over SparseVLM and +1.6% over VisPruner; 96.9% retained at 80.2% pruning; Qwen2.5-VL gains 0.5 on MathVista at ~60.5% reduction; LLaVA-OneVision-7B reaches 96.2% with only 15% of tokens vs. 94.6% for FrameFusion.