AutoEncoder-Compressed Parallel Split Learning for Pre-trained Model Fine-Tuning
Merged summary
TL;DR - AE-PSL is a split-learning framework that uses a lightweight autoencoder to reduce activation and gradient communication when fine-tuning foundation models across constrained edge devices. Its two-stage alignment aims to preserve compatibility with pretrained and client-specific feature distributions.
- Keeps only a few model layers on clients while offloading most computation to a server.
- Compresses intermediate activations and gradients at the model’s split layer.
- Aligns the autoencoder with the pretrained model’s feature manifold before distributed fine-tuning.
- Further adapts compression to client-specific feature distributions to mitigate representation misalignment.
Sources (1)
AutoEncoder-Compressed Parallel Split Learning for Pre-trained Model Fine-Tuning
TL;DR - AE-PSL is a split-learning framework that uses a lightweight autoencoder to reduce activation and gradient communication when fine-tuning foundation models across constrained edge devices. Its two-stage alignment aims to preserve compatibility with pretrained and client-specific feature distributions.
- Keeps only a few model layers on clients while offloading most computation to a server.
- Compresses intermediate activations and gradients at the model’s split layer.
- Aligns the autoencoder with the pretrained model’s feature manifold before distributed fine-tuning.
- Further adapts compression to client-specific feature distributions to mitigate representation misalignment.