Mitigating Reasoning-Induced Misalignment via Safety-Direction Penalty
Ranking
Overall
78
Content
95
Popularity
37
Observed public metrics from 1 member.
Merged summary
TL;DR - This paper analyzes how harmless reasoning fine-tuning can sometimes degrade LLM safety and introduces a Safety-Direction Penalty (SDP) to mitigate that effect. On Qwen2.5-3B and 7B, SDP restores safety while preserving benchmark reasoning performance.
- Identifies coupled activation-space directions associated with reasoning ability and safety behavior.
- Finds that larger safety-representation shifts correlate with greater safety degradation.
- Uses CKA distance ratios and probes to locate layers most relevant to safety decisions.
- Penalizes movement along the learned safety direction during fine-tuning, expanding the targeted layers when diagnostics reveal compensatory shifts elsewhere.
Sources (1)
Mitigating Reasoning-Induced Misalignment via Safety-Direction Penalty
Public signals
Semantic Scholar citations 0 · Semantic Scholar influential citations 0
TL;DR - This paper analyzes how harmless reasoning fine-tuning can sometimes degrade LLM safety and introduces a Safety-Direction Penalty (SDP) to mitigate that effect. On Qwen2.5-3B and 7B, SDP restores safety while preserving benchmark reasoning performance.
- Identifies coupled activation-space directions associated with reasoning ability and safety behavior.
- Finds that larger safety-representation shifts correlate with greater safety degradation.
- Uses CKA distance ratios and probes to locate layers most relevant to safety decisions.
- Penalizes movement along the learned safety direction during fine-tuning, expanding the targeted layers when diagnostics reveal compensatory shifts elsewhere.