Multimodal Semantic-Aware Contrastive Learning For False Negative Mitigation in 3D Medical Imaging

TL;DR - MseaCL is a multimodal contrastive learning framework for 3D pediatric brain MRI and radiology reports that reduces false negatives by using semantic similarity between reports as a guiding signal, yielding large downstream gains in clinical tasks.

• Targets a known flaw in standard contrastive learning: treating all non-paired samples as negatives creates false negatives when samples share high-level semantic attributes, common in medical data.
• Uses semantic similarity between radiology reports to guide alignment during pretraining, rather than assuming strict instance-based correspondence within a batch.
• Trained on a pediatric cohort of 3D brain MRI scans paired with radiology reports.
• Reports at least a 22.6% AUC increase on pediatric brain tumor molecular classification when used as a pretraining stage.