Not All Eval-Awareness Is Equal: Capabilities Framing Predicts Compliance
Ranking
Overall
74
Content
90
Popularity
37
Observed public metrics from 1 member.
Merged summary
TL;DR - This paper finds that eval-awareness is not a behaviorally uniform signal: whether a model frames testing as a capabilities check or a safety-boundary check strongly predicts compliance. This matters because aggregate suppression metrics may obscure whether safety-relevant awareness actually changed.
- On Qwen3-32B and FORTRESS, capabilities-framed awareness predicted 24–46 percentage points more compliance than safety-framed awareness across all steering conditions.
- Chain-of-thought eval-awareness was classified as capabilities-flavored, safety-flavored, both, or neither.
- CoT-prefill experiments shifted compliance in the predicted direction for 10 of 11 prefills, suggesting a causal relationship.
- Identical aggregate eval-awareness suppression rates can correspond to qualitatively different safety outcomes.
Sources (1)
Not All Eval-Awareness Is Equal: Capabilities Framing Predicts Compliance
Public signals
Semantic Scholar citations 0 · Semantic Scholar influential citations 0
TL;DR - This paper finds that eval-awareness is not a behaviorally uniform signal: whether a model frames testing as a capabilities check or a safety-boundary check strongly predicts compliance. This matters because aggregate suppression metrics may obscure whether safety-relevant awareness actually changed.
- On Qwen3-32B and FORTRESS, capabilities-framed awareness predicted 24–46 percentage points more compliance than safety-framed awareness across all steering conditions.
- Chain-of-thought eval-awareness was classified as capabilities-flavored, safety-flavored, both, or neither.
- CoT-prefill experiments shifted compliance in the predicted direction for 10 of 11 prefills, suggesting a causal relationship.
- Identical aggregate eval-awareness suppression rates can correspond to qualitatively different safety outcomes.