Toward Better Assessment of LLMs' Performance in Clinical Error Detection

TL;DR - Standard F1-style metrics can substantially misrepresent LLM performance on clinical error detection. Paired evaluation reveals that 13 of 15 tested models performed below random pairwise discrimination, raising concerns for safety-critical deployment.

• Evaluated 15 LLMs on four standardized test sets spanning three languages.
• Models often found error-relevant text but failed to correctly distinguish erroneous notes from their clean counterparts.
• Error-labeling biases varied by language, from defaulting to “no error” to over-flagging errors.
• F1 and pairwise accuracy responded oppositely to these biases, potentially causing F1 rankings to favor weak discriminators.