Benchmarking biomedical foundation models

TL;DR - This Nature Methods Perspective examines shortcomings in how biomedical foundation models are benchmarked and proposes guidelines for more rigorous, meaningful evaluation. Better benchmarks are needed to assess these models’ capabilities reliably across biological domains.

• Foundation models are gaining adoption across diverse areas of biology.
• Existing evaluation methods do not comprehensively capture their full potential.
• The article highlights current limitations and challenges in biomedical model benchmarking.
• It proposes guidance for designing more robust and informative evaluations.