Improving O-RADS Risk Stratification from Ultrasound Reports: A Comparative Evaluation of Hybrid versus End-to-End LLM Reasoning Strategies

TL;DR - A retrospective study found that a hybrid architecture—using an LLM to extract ultrasound features and deterministic rules to assign O-RADS categories—outperformed end-to-end LLM reasoning and original clinical reports. The approach matters because it improved accuracy while making guideline-based decisions more reliable and interpretable.

• Eight LLMs and three reasoning strategies were evaluated on 390 ovarian masses from 310 patients.
• Gemini 3.6 Flash with the hybrid strategy achieved 99.2% accuracy and a weighted kappa of 1.00 against expert consensus.
• End-to-end strategies achieved 65.6%–95.9% accuracy, while original clinical reports achieved 87.7%.
• Separating feature extraction from rule execution reduced classification errors and mitigated overstaging.