LoRA-TSD: Tangent-Space Spectral Descent for LoRA via Muon-Style Updates

TL;DR - LoRA-TSD is a geometry-aware optimizer that applies Muon-style spectral descent within the tangent space of the fixed-rank LoRA update. It improves benchmark performance across several model scales while reducing retraction cost and providing convergence guarantees.

• Treats each LoRA update as a tangent vector on a fixed-rank matrix manifold rather than optimizing its two factors independently.
• Uses a LoRA-native retraction that is up to 2.8× cheaper than the truncated-SVD retraction used in prior manifold methods.
• Establishes the tangent-projected gradient as a stationarity measure and gives the first global convergence guarantees under this measure for LoRA-TSD and LoRA-Pro.
• Outperforms competing LoRA optimizers across six commonsense and natural-language-inference benchmarks using Llama and Qwen models, while remaining robust to adapter rank.