HarMoE: Multi-Source Chest Radiograph Pretraining with Dataset-Disentangled Experts

TL;DR - HarMoE is a chest X-ray vision-language pretraining framework that harmonizes many heterogeneous multi-label classification datasets instead of relying mainly on MIMIC-CXR image-report pairs, using dataset-aware mixture-of-experts to keep clinical semantics separate from dataset identity. It matters because it shows scaling radiology VLMs can come from cleaner, broader labeled supervision rather than more free-text reports.

• Core problem: differences in label ontologies, annotation protocols, acquisition pipelines, and report styles cause models to entangle clinical semantics with dataset identity, hurting transfer even as data scale grows.
• Method: a shared backbone learns cross-dataset medical semantics while source-specific variation is confined to lightweight residual experts placed in deeper decoder layers.
• Supervision: training uses a unified disease vocabulary with masked multi-dataset supervision, so complementary annotations across sources can be combined without creating false negatives.
• Reported gains over strong baselines on zero-shot classification, out-of-distribution transfer, and grounding; code plus an 873k-image harmonized dataset are slated for release at github.com/Roypic/harmoe.