‹ back to 2026-07-27

\k{appa}-LoRA: Condition Numbers Reveal Which LoRA Matrices Worth Updating

Research Efficiency & Systems

Merged summary

TL;DR - κ-LoRA selectively updates LoRA matrices with the highest condition numbers, preserving standard LoRA accuracy while reducing fine-tuning costs.

Updating only the top 50% of matrices halves the trainable parameter count.
Experiments report 16.2% lower fine-tuning time and 4.5% lower memory usage on average.
High-condition-number matrices contain underdeveloped directions that contribute most to adaptation.
Their condition numbers decline during training, suggesting targeted spectral rebalancing drives the gains.

Sources (1)

\k{appa}-LoRA: Condition Numbers Reveal Which LoRA Matrices Worth Updating

arXiv cs.LG Jianghui Wang, Silong Yong, Francesco Orabona, Marco Canini, Katia P. Sycara, Yaqi Xie 2026-07-24 arXiv:2607.22489

TL;DR - κ-LoRA selectively updates LoRA matrices with the highest condition numbers, preserving standard LoRA accuracy while reducing fine-tuning costs.

Updating only the top 50% of matrices halves the trainable parameter count.
Experiments report 16.2% lower fine-tuning time and 4.5% lower memory usage on average.
High-condition-number matrices contain underdeveloped directions that contribute most to adaptation.
Their condition numbers decline during training, suggesting targeted spectral rebalancing drives the gains.

item →