Mitigating Reasoning-Induced Misalignment via Safety-Direction Penalty

TL;DR - This paper analyzes how harmless reasoning fine-tuning can sometimes degrade LLM safety and introduces a Safety-Direction Penalty (SDP) to mitigate that effect. On Qwen2.5-3B and 7B, SDP restores safety while preserving benchmark reasoning performance.

• Identifies coupled activation-space directions associated with reasoning ability and safety behavior.
• Finds that larger safety-representation shifts correlate with greater safety degradation.
• Uses CKA distance ratios and probes to locate layers most relevant to safety decisions.
• Penalizes movement along the learned safety direction during fine-tuning, expanding the targeted layers when diagnostics reveal compensatory shifts elsewhere.