Autoregressive EHR Foundation Models with Multimodal Inputs
Merged summary
TL;DR - A multimodal autoregressive EHR foundation-model framework integrates ECGs, chest X-rays, and clinical notes using latent compression and temporally aligned gated cross-attention. Results show fusion design and encoder quality matter, while adding modalities alone does not necessarily improve mortality prediction.
- Effective latent compression outperformed uncompressed cross-attention and mean pooling on MIMIC-IV.
- Stronger pretrained modality encoders consistently improved downstream performance within each modality.
- Auxiliary modalities did not automatically outperform the EHR-only baseline for ICU mortality prediction.
- Careful fusion architecture and clinically appropriate evaluation are essential.
Sources (1)
Autoregressive EHR Foundation Models with Multimodal Inputs
TL;DR - A multimodal autoregressive EHR foundation-model framework integrates ECGs, chest X-rays, and clinical notes using latent compression and temporally aligned gated cross-attention. Results show fusion design and encoder quality matter, while adding modalities alone does not necessarily improve mortality prediction.
- Effective latent compression outperformed uncompressed cross-attention and mean pooling on MIMIC-IV.
- Stronger pretrained modality encoders consistently improved downstream performance within each modality.
- Auxiliary modalities did not automatically outperform the EHR-only baseline for ICU mortality prediction.
- Careful fusion architecture and clinically appropriate evaluation are essential.