‹ back to 2026-07-24

Adaptive Depth Sparse Framework: Similarity-Driven Resource Allocation for Pre-Trained LLMs

arXiv cs.CL Efficiency & Systems Yidu Wu, Xiang Wang, Kejie Zhao, Zhangchi Wang, Qinghai Guo, Xiaoying Tang 2026-07-23

TL;DR - AdaDSF converts pretrained LLMs into depth-sparse models without full retraining, reducing inference FLOPs while retaining near-dense performance.

Uses input-output cosine similarity to estimate each layer’s contribution and allocate token retention ratios.
Employs lightweight routers to select informative tokens at each layer.
Aligns sparse and dense models’ intermediate and final representations to preserve features.
On GPT-NeoX and Qwen2.5, it outperformed MoD, D-LLM, and DLO in accuracy retention at comparable sparsity.

view merged work →