MedFailBench: A Clinician-Built Open-Source Benchmark for Medical AI Safety Boundary Inspection

TL;DR - MedFailBench is a clinician-built, open-source benchmark that shifts medical AI evaluation from "did the model get the right answer?" to "which safety boundary failed?"—labeling errors by severity and failure type, which matters for surfacing dangerous, high-stakes model behaviors rather than just accuracy.

• Introduces a safety-gate taxonomy of failure modes: missed urgent escalation, unsafe remote dosing, unsafe discharge reassurance, evidence fabrication, unsafe protocol execution, and source support gap, plus a 1–5 clinical severity rubric.
• Current public release (v0.2.1) is small: 44 clinician-reviewed synthetic cases with severity annotations—no patient data, no clinical validation claims, and no model rankings.
• Ships with a live HuggingFace leaderboard preview and an automated pipeline for archiving model-response screening runs.
• Openly licensed (Apache-2.0 and CC-BY-4.0) with a Zenodo DOI; note the benchmark is synthetic and early-stage, so takeaways are about methodology/framing rather than empirical results.