Inadvertent Context Leakage in Language Models

TL;DR - This paper shows that language models can inadvertently encode sensitive in-context information into benign outputs, enabling black-box attackers to reconstruct secrets even when direct extraction is refused. The findings expose a serious privacy risk for agents handling personal data.

• Across eight proprietary models, attacks recovered 2-digit secrets with near-perfect accuracy and 4-digit secrets with 82% exact match from ordinary, non-adversarial responses.
• More capable models leaked more information, suggesting the effect may arise from stronger instruction-following rather than a narrowly patchable flaw.
• A trained classifier inferred health and financial predicates about user memories from routine natural-language outputs.
• An RL-trained adversary extracted complete Social Security Numbers from a production-style agent through seemingly innocuous text.