HarMoE: Multi-Source Chest Radiograph Pretraining with Dataset-Disentangled Experts
Ranking
Observed public metrics from 1 member.
Merged summary
TL;DR - HarMoE is a chest X-ray vision-language pretraining framework that harmonizes many heterogeneous multi-label classification datasets instead of relying mainly on MIMIC-CXR image-report pairs, using dataset-aware mixture-of-experts to keep clinical semantics separate from dataset identity. It matters because it shows scaling radiology VLMs can come from cleaner, broader labeled supervision rather than more free-text reports.
- Core problem: differences in label ontologies, annotation protocols, acquisition pipelines, and report styles cause models to entangle clinical semantics with dataset identity, hurting transfer even as data scale grows.
- Method: a shared backbone learns cross-dataset medical semantics while source-specific variation is confined to lightweight residual experts placed in deeper decoder layers.
- Supervision: training uses a unified disease vocabulary with masked multi-dataset supervision, so complementary annotations across sources can be combined without creating false negatives.
- Reported gains over strong baselines on zero-shot classification, out-of-distribution transfer, and grounding; code plus an 873k-image harmonized dataset are slated for release at github.com/Roypic/harmoe.
Sources (1)
HarMoE: Multi-Source Chest Radiograph Pretraining with Dataset-Disentangled Experts
TL;DR - HarMoE is a chest X-ray vision-language pretraining framework that harmonizes many heterogeneous multi-label classification datasets instead of relying mainly on MIMIC-CXR image-report pairs, using dataset-aware mixture-of-experts to keep clinical semantics separate from dataset identity. It matters because it shows scaling radiology VLMs can come from cleaner, broader labeled supervision rather than more free-text reports.
- Core problem: differences in label ontologies, annotation protocols, acquisition pipelines, and report styles cause models to entangle clinical semantics with dataset identity, hurting transfer even as data scale grows.
- Method: a shared backbone learns cross-dataset medical semantics while source-specific variation is confined to lightweight residual experts placed in deeper decoder layers.
- Supervision: training uses a unified disease vocabulary with masked multi-dataset supervision, so complementary annotations across sources can be combined without creating false negatives.
- Reported gains over strong baselines on zero-shot classification, out-of-distribution transfer, and grounding; code plus an 873k-image harmonized dataset are slated for release at github.com/Roypic/harmoe.