‹ back to 2026-07-24

Adaptive Depth Sparse Framework: Similarity-Driven Resource Allocation for Pre-Trained LLMs

Research Efficiency & Systems

Ranking

Overall 65

Content 75

Popularity 42

Observed public metrics from 1 member.

Merged summary

TL;DR - AdaDSF converts pretrained LLMs into depth-sparse models without full retraining, reducing inference FLOPs while retaining near-dense performance.

Uses input-output cosine similarity to estimate each layer’s contribution and allocate token retention ratios.
Employs lightweight routers to select informative tokens at each layer.
Aligns sparse and dense models’ intermediate and final representations to preserve features.
On GPT-NeoX and Qwen2.5, it outperformed MoD, D-LLM, and DLO in accuracy retention at comparable sparsity.

Sources (1)

Adaptive Depth Sparse Framework: Similarity-Driven Resource Allocation for Pre-Trained LLMs

arXiv cs.CL Yidu Wu, Xiang Wang, Kejie Zhao, Zhangchi Wang, Qinghai Guo, Xiaoying Tang 2026-07-23 arXiv:2607.21291 doi:10.1007/978-981-92-3447-9_43

Public signals OpenAlex citations 0 · Semantic Scholar citations 0 · Semantic Scholar influential citations 0

Providers: Hugging Face · N/A OpenAlex · Citations 0 Publisher · N/A Semantic Scholar · Citations 0 · Influential citations 0 X · N/A Fetched 2026-08-23 14:28:53.826012 UTC

TL;DR - AdaDSF converts pretrained LLMs into depth-sparse models without full retraining, reducing inference FLOPs while retaining near-dense performance.

Uses input-output cosine similarity to estimate each layer’s contribution and allocate token retention ratios.
Employs lightweight routers to select informative tokens at each layer.
Aligns sparse and dense models’ intermediate and final representations to preserve features.
On GPT-NeoX and Qwen2.5, it outperformed MoD, D-LLM, and DLO in accuracy retention at comparable sparsity.

item →