Self-Play Pretraining with Zero Data

TL;DR - This paper presents a zero-data pretraining method in which a generator and learner co-evolve from random initialization, creating an adaptive synthetic curriculum without natural training data. It matters because zero-shot performance on natural datasets improves predictably with self-play compute, suggesting a potential route to pretraining limited more by computation than curated data.

• A generator uses reinforcement learning to propose programs for a universal Turing machine, which produces byte sequences for training.
• An autoregressive learner predicts those sequences with standard cross-entropy, while the generator targets the frontier of the learner’s capabilities.
• Zero-shot loss on several natural datasets follows predictable compute scaling despite neither model seeing natural data during training.
• The learner develops in-context learning behavior, and the system discovers recognizable mathematical sequences.