How Many Bits Can an Adapter Write? Measuring the Capacity and Memorization of Parameter-Efficient Fine-Tuning

TL;DR - This paper quantifies how much information LoRA adapters memorize when fine-tuning frozen models. It finds that capacity and privacy leakage depend more on adapter placement and base-model structure than raw parameter count.

• Adapters store a few bits per trainable parameter, less than full fine-tuning.
• Moving an equal parameter budget from attention to MLP layers nearly doubles capacity.
• Capacity largely disappears when the frozen base model’s structure is removed.
• In Qwen2.5 experiments, leakage tracks written bits; supervised tuning memorizes secrets verbatim, while training on verifiable rewards does not.