Autoregressive EHR Foundation Models with Multimodal Inputs

TL;DR - A multimodal autoregressive EHR foundation-model framework integrates ECGs, chest X-rays, and clinical notes using latent compression and temporally aligned gated cross-attention. Results show fusion design and encoder quality matter, while adding modalities alone does not necessarily improve mortality prediction.

• Effective latent compression outperformed uncompressed cross-attention and mean pooling on MIMIC-IV.
• Stronger pretrained modality encoders consistently improved downstream performance within each modality.
• Auxiliary modalities did not automatically outperform the EHR-only baseline for ICU mortality prediction.
• Careful fusion architecture and clinically appropriate evaluation are essential.