A Multimodal Foundation Model for Longitudinal Patient Representation and Scalable Insight Generation in Oncology

TL;DR - oFM is a multimodal foundation model trained on longitudinal records from more than one million oncology patients, integrating clinical history, genomics, transcriptomics, and pathology images. Its frozen patient embeddings substantially improved prognostic and treatment-benefit predictions over expert-curated baseline features.

• The model encodes daily clinical and molecular episodes, combines them with H&E pathology images, and integrates observations over time into patient-state embeddings.
• Training used patient-level partitions from a real-world cohort of 1.67 million cancer patients.
• oFM improved overall-survival AUC from 0.563 to 0.774 and also improved treatment-response and progression-free-survival prediction.
• Across 11 comparative-treatment cohorts, its embeddings achieved three-fold higher pooled, scale-normalized treatment-benefit AUTOC and improved benefit ranking in 9 cohorts.