BioM-JEPA: joint-embedding prediction of graph-connected gene blocks in single cells

TL;DR - BioM-JEPA is a self-supervised single-cell transcriptomics model that, instead of reconstructing individual genes, predicts joint embeddings of graph-connected gene blocks derived from protein-association and coexpression evidence. It matters because block-level prediction yields richer, less depth-confounded cell representations while being substantially cheaper to train and serve.

• Architecture: A student network predicts a target gene-block representation from the remaining genes in a cell; a slowly updated teacher produces the target from the full observed gene set (JEPA-style). Gene blocks come from protein-association plus corpus-derived coexpression graphs.
• Representation quality: Under the reported extraction procedure, embeddings showed higher effective rank and weaker association with detected-gene depth than token-prediction, random-block, and reconstruction controls.
• Downstream results: Frozen embeddings retained expression, pathway, and neighbourhood information across CellBench tasks and achieved the lowest aggregate perturbation-response error among evaluated models; diagnostics aligned with canonical pancreatic programmes and perturbation compositionality.
• Efficiency: Linear attention avoids a quadratic gene-by-gene attention matrix — in a matched one-epoch hPancreas run at batch size 8, it gave 5.75× higher fine-tuning throughput and 3.76× higher held-out embedding throughput than scFoundation.