Is Self-Pretraining really useful to improve diagnosis in medical Time Series?

TL;DR - An arXiv study testing whether Self-PreTraining (SPT) with masking objectives helps transformers on medical time-series classification, finding consistent but modest gains across three clinical datasets. It matters because SPT is an architecture-agnostic way to squeeze more accuracy out of data-limited clinical settings.

• Evaluated transformers on three tasks: rehabilitation robotics (Camargo), stress detection (Non-EEG Stress), and Parkinson's disease detection (Gait PD), training either from scratch or via SPT.
• Four masking-based pre-training objectives targeting temporal and cross-modal representation learning yielded 0–6 percentage point accuracy improvements, varying by masking strategy, dataset, and architecture.
• Gains held not only for multimodal/multivariate inputs but also for simple univariate inputs, suggesting the benefit is not purely cross-modal.
• Model depth was varied systematically; deeper models benefited more, indicating capacity is needed to exploit the pre-trained temporal representations.