Decodable but Misrouted: Sparse Features Uncover a Readout Gap in Vision-Language Models for Harmful Meme Detection

TL;DR - Sparse autoencoder analysis suggests vision-language models often encode evidence needed to detect harmful memes but fail to route it into their final predictions. Improved calibration and targeted LoRA adaptation can recover much of this readout gap.

• Sparse probes substantially outperformed native macro-F1 across six binary benchmarks: 0.740 vs. 0.432 for Qwen and 0.714 vs. 0.532 for Gemma.
• Causal ablation and feature-patching experiments distinguished internally represented “silent” evidence from evidence already routed toward model outputs.
• Calibration-only routing recovered 93.3% of the mean performance gap, while probe-distilled LoRA also improved native predictions; shared multi-task adaptation caused negative transfer.
• Tests in Spanish and Hindi-English code-mixed settings indicated that the signal extends beyond English, requires paired visual evidence, and is not solely explained by OCR.