Normalized Low-Rank Adaptation

TL;DR - Normalized Low-Rank Adaptation (NoRA) normalizes LoRA’s down-projection matrices to stabilize optimization. It reportedly improves convergence, performance, and retention without adding parameters or inference-time computation.

• Targets LoRA’s early training dynamics, which are dominated by the down-projection because the up-projection is initialized to zero.
• Supports either continuous normalization during training or a one-time normalization at initialization.
• Shows consistent benefits across pretraining, supervised fine-tuning, and reinforcement learning.
• Mitigates catastrophic forgetting while preserving LoRA’s parameter and inference efficiency.