Probing Speaker Identity Sensitivity in Audio Deepfake Detectors

TL;DR - This paper introduces the Identity Sensitivity Score (ISS), an inference-time diagnostic for detecting when audio deepfake classifiers rely on speaker identity rather than synthesis artifacts. ISS identifies likely errors without ground-truth labels and helps explain poor cross-dataset generalization.

• Misclassified utterances had ISS values 29–52 times higher than correctly classified ones.
• ISS predicted misclassification with AUC as high as 0.954 across two detectors and datasets.
• Voice-conversion tests showed ISS-flagged utterances produced 19–30 times larger detector-score shifts than stable utterances.
• ISS requires only detector scores and a reference pool of speaker examples.