The Blessing of Dimensionality: How Near-Orthogonality in High-Dimensional Spaces Explains Temporal Portability
Merged summary
TL;DR - PortLLM’s LoRA patches remain effective across 10 continual-pretraining updates on multiple model families. The paper attributes this temporal portability to near-orthogonality in high-dimensional parameter spaces.
- Evaluates Mistral, Gemma, and Qwen across 10 continual-pretraining steps.
- Finds that repeated fine-tuning may be unnecessary after periodic base-model updates.
- Provides two theoretical analyses explaining PortLLM’s competitive performance.
- Uses loss-landscape geometry to compare model-adaptation options.
Sources (1)
The Blessing of Dimensionality: How Near-Orthogonality in High-Dimensional Spaces Explains Temporal Portability
TL;DR - PortLLM’s LoRA patches remain effective across 10 continual-pretraining updates on multiple model families. The paper attributes this temporal portability to near-orthogonality in high-dimensional parameter spaces.
- Evaluates Mistral, Gemma, and Qwen across 10 continual-pretraining steps.
- Finds that repeated fine-tuning may be unnecessary after periodic base-model updates.
- Provides two theoretical analyses explaining PortLLM’s competitive performance.
- Uses loss-landscape geometry to compare model-adaptation options.