Necessary or Sufficient? Evaluating LLM Explanations With Behavioural Evidence

TL;DR - This paper introduces a black-box intervention framework for testing whether factors cited in LLM explanations are necessary or sufficient for model decisions. Across eight Claude, GPT, and Gemini models, cited factors were informative but only moderately aligned with measured behavioral influence, limiting their reliability for agent oversight.

• Controlled interventions measured necessity by changing a cited factor and sufficiency by retaining it while removing other changeable information.
• Mean rank correlations ranged from 0.349 to 0.580 across advisor-recommendation and prompt-monitoring tasks.
• In advisor recommendations, an uncited factor outperformed the weakest cited factor in roughly 58% of responses under both measures.
• Prompt monitoring showed better alignment: the corresponding rates were 25.8% for necessity and 8.9% for sufficiency.