Spectral Transformation for Layer-wise Global Rank Discovery in Federated LoRA for Vision Transformers

TL;DR - SpecTraL improves federated LoRA fine-tuning of Vision Transformers by aggregating adapters in low-rank latent space and automatically selecting global ranks per layer. It aims to reduce communication and server computation while improving convergence under heterogeneous client data.

• Uses orthonormal Householder transformations to avoid mathematically inconsistent factor averaging and dense update reconstruction.
• Applies a spiked covariance model to distinguish shared signal from non-IID noise and determine layer-wise ranks without hyperparameter search.
• Introduces padding-aware initialization so clients can adopt residual LoRA dimensions without merging updates into base-model weights.
• Experiments on ViT-B/16 and ViT-L/16 with DomainNet and NICO++ show improved accuracy–communication trade-offs and lower server overhead.