Nat. Med. | 经临床验证的人工智能聊天机器人心理健康交互行为审计框架
Ranking
Observed public metrics from 1 member.
Merged summary
TL;DR - A Nature Medicine paper introduces SIM-VAIL, a clinically validated automated red-teaming framework that audits AI chatbots through simulated multi-turn conversations with psychologically vulnerable users, showing that mental-health risk emerges cumulatively from dialogue dynamics rather than from single harmful replies. It matters because standard single-turn safety benchmarks systematically miss these interactional harms.
- Design & scale: 5 vulnerability states × 6 interaction intents = 30 clinically grounded simulated personas, run against 9 frontier chatbots (3 repeats, up to 10 turns) → 810 conversations, 6,329 turns, scored by an automated judge on 39 behavioral dimensions including 13 prespecified mental-health risk dimensions.
- Validation: inter-judge correlation r = 0.91; ICC 0.90 across repeats; median AUC 0.98 separating known high- vs low-risk dialogues; 27 clinicians rated 488 turns, with judge–human agreement exceeding human–human agreement; mean realism 4.15/5.
- Key finding (VAIL): risk is not a fixed model property but a function of vulnerability × intent × model × trajectory. Psychotic and manic vulnerability, plus glamorization, emotional-dependence, and dangerous-behavior intents, scored highest; clustering yielded four trajectories (low, gradual escalation, early escalation, recovery). PC1 explained 62.4% of variance, contrasting therapeutic quality against concerning behavior/sycophancy/belief reinforcement.
- Intervention: counterfactual rewriting of a single user message or the chatbot's first high-risk reply at the inflection point significantly lowered downstream risk, with effects still detectable five turns later — supporting real-time, message-level safety layers.
- Limits: simulated users only (30 personas), raw API models rather than deployed consumer products with system prompts/safety middleware; authors frame results as a clinically meaningful risk lower bound, not a diagnostic tool. Under the study's specific API versions, Claude Sonnet 4.5 scored lowest and Grok 4 highest on overall concerning behavior.
Sources (1)
Nat. Med. | 经临床验证的人工智能聊天机器人心理健康交互行为审计框架
TL;DR - A Nature Medicine paper introduces SIM-VAIL, a clinically validated automated red-teaming framework that audits AI chatbots through simulated multi-turn conversations with psychologically vulnerable users, showing that mental-health risk emerges cumulatively from dialogue dynamics rather than from single harmful replies. It matters because standard single-turn safety benchmarks systematically miss these interactional harms.
- Design & scale: 5 vulnerability states × 6 interaction intents = 30 clinically grounded simulated personas, run against 9 frontier chatbots (3 repeats, up to 10 turns) → 810 conversations, 6,329 turns, scored by an automated judge on 39 behavioral dimensions including 13 prespecified mental-health risk dimensions.
- Validation: inter-judge correlation r = 0.91; ICC 0.90 across repeats; median AUC 0.98 separating known high- vs low-risk dialogues; 27 clinicians rated 488 turns, with judge–human agreement exceeding human–human agreement; mean realism 4.15/5.
- Key finding (VAIL): risk is not a fixed model property but a function of vulnerability × intent × model × trajectory. Psychotic and manic vulnerability, plus glamorization, emotional-dependence, and dangerous-behavior intents, scored highest; clustering yielded four trajectories (low, gradual escalation, early escalation, recovery). PC1 explained 62.4% of variance, contrasting therapeutic quality against concerning behavior/sycophancy/belief reinforcement.
- Intervention: counterfactual rewriting of a single user message or the chatbot's first high-risk reply at the inflection point significantly lowered downstream risk, with effects still detectable five turns later — supporting real-time, message-level safety layers.
- Limits: simulated users only (30 personas), raw API models rather than deployed consumer products with system prompts/safety middleware; authors frame results as a clinically meaningful risk lower bound, not a diagnostic tool. Under the study's specific API versions, Claude Sonnet 4.5 scored lowest and Grok 4 highest on overall concerning behavior.