Statistical Inference for Rank Allocation in Low-Rank Adaptation

TL;DR - StatLoRA frames rank allocation for parameter-efficient fine-tuning as statistical hypothesis testing, using estimated p-values to retain or prune LoRA components. It matches or outperforms several LoRA baselines under equal rank budgets.

• Establishes asymptotic normality for stochastic optimizer trajectories, including AdamW.
• Derives distributions for component-level test statistics used in rank allocation.
• Evaluates DeBERTaV3-base, BART-Large, and Qwen2.5-7B across understanding, generation, and question-answering tasks.
• Reports stable allocation behavior and empirical support for the proposed asymptotic theory.