OrthKD: Extracting Generalized Clinical Knowledge from Heterogeneous Teachers for Lightweight Deployment

TL;DR - OrthKD selectively distills complementary knowledge from heterogeneous teachers into a lightweight diabetic-retinopathy screening model, improving edge deployment and robustness under domain shift.

• Uses full supervision from a stronger EfficientNet-B3 teacher but only feature-level supervision from a weaker Swin-Base teacher.
• Orthogonal student projections encourage teacher-specific features to contribute complementary local and global evidence.
• The 5.4M-parameter MobileNetV3 student achieves 0.885 QWK on EyePACS.
• Zero-shot Messidor-2 performance rises from 0.507 to 0.728 QWK, alongside strong referral AUC and calibration.