Training Large Language Models for Self-Explanation Faithfulness

TL;DR - This paper uses reinforcement learning to train LLMs to produce self-explanations that better reflect factors influencing their decisions. It offers a potential scalable route to more faithful and transparent model reasoning.

• Converts the Phi-CCT faithfulness metric into a per-sample RL reward.
• Fine-tuned Llama3.1-8B and Qwen3-8B achieve Phi-CCT scores up to 0.664 in-distribution and 0.691 on held-out tasks.
• Experiments test whether models disclose the effects of random-word and user-bias interventions.
• Cross-intervention transfer is limited and model-dependent, while analyses found no evident reward gaming.