Where Identity Lives: Localized, Retain-Free Identity Unlearning in Multimodal Large Language Models

TL;DR - PAVA is a retain-free method for removing an individual’s identity information from multimodal large language models while preserving visual-language capabilities. It matters because it avoids reconstructing privacy-sensitive retain sets after deployment.

• Causal tracing, weight transplant, and Fisher overlap localize identity information primarily to early-to-mid decoder MLP layers.
• PAVA restricts updates to those layers to reduce disruption to shared visual-language processing.
• A visual-attribute anchor distills the model’s pre-unlearning, image-grounded answers using only forget-set images.
• On MLLMU-Bench and ReMem, PAVA achieves the strongest forget-retain trade-off among forget-set-only methods and remains competitive with retain-based baselines.