The Blessing of Dimensionality: How Near-Orthogonality in High-Dimensional Spaces Explains Temporal Portability

TL;DR - PortLLM’s LoRA patches remain effective across 10 continual-pretraining updates on multiple model families. The paper attributes this temporal portability to near-orthogonality in high-dimensional parameter spaces.

• Evaluates Mistral, Gemma, and Qwen across 10 continual-pretraining steps.
• Finds that repeated fine-tuning may be unnecessary after periodic base-model updates.
• Provides two theoretical analyses explaining PortLLM’s competitive performance.
• Uses loss-landscape geometry to compare model-adaptation options.