Spectral Transformation for Layer-wise Global Rank Discovery in Federated LoRA for Vision Transformers
Merged summary
TL;DR - SpecTraL improves federated LoRA fine-tuning of Vision Transformers by aggregating adapters in low-rank latent space and automatically selecting global ranks per layer. It aims to reduce communication and server computation while improving convergence under heterogeneous client data.
- Uses orthonormal Householder transformations to avoid mathematically inconsistent factor averaging and dense update reconstruction.
- Applies a spiked covariance model to distinguish shared signal from non-IID noise and determine layer-wise ranks without hyperparameter search.
- Introduces padding-aware initialization so clients can adopt residual LoRA dimensions without merging updates into base-model weights.
- Experiments on ViT-B/16 and ViT-L/16 with DomainNet and NICO++ show improved accuracy–communication trade-offs and lower server overhead.
Sources (1)
Spectral Transformation for Layer-wise Global Rank Discovery in Federated LoRA for Vision Transformers
TL;DR - SpecTraL improves federated LoRA fine-tuning of Vision Transformers by aggregating adapters in low-rank latent space and automatically selecting global ranks per layer. It aims to reduce communication and server computation while improving convergence under heterogeneous client data.
- Uses orthonormal Householder transformations to avoid mathematically inconsistent factor averaging and dense update reconstruction.
- Applies a spiked covariance model to distinguish shared signal from non-IID noise and determine layer-wise ranks without hyperparameter search.
- Introduces padding-aware initialization so clients can adopt residual LoRA dimensions without merging updates into base-model weights.
- Experiments on ViT-B/16 and ViT-L/16 with DomainNet and NICO++ show improved accuracy–communication trade-offs and lower server overhead.