Pretraining EHR Foundation Models with Patient-Aware Sampling

TL;DR - This paper proposes patient-aware sampling for pretraining autoregressive EHR foundation models, avoiding mixed-patient windows and controlling each patient’s training contribution. It improves downstream clinical performance over conventional global-stream sampling.

• Standard token-stream sampling can overrepresent patients with longer records.
• Stochastic Patient Sampling supports controllable weighting of training signals across patients.
• It improves Macro AUROC and AUPRC on clinical tasks using MIMIC-IV v2.2 and v3.1.
• Sequence construction is identified as an important, underexplored EHR pretraining design choice.