Puro-2B: Poor Lab's Qwen2-1.5B Trained on RTX 5090 within $5090

TL;DR - Puro-2B is an open recipe for pretraining 2B-parameter language models from scratch on consumer RTX 5090 GPUs, reaching near-Qwen2.5-1.5B performance for under $6,900. It aims to make controlled LLM pretraining research substantially more accessible.

• Models were trained with FP8 precision on as many as 1.4 trillion tokens.
• Cost reductions combine consumer hardware, low-precision training, hyperball optimization, curriculum model averaging, and a tailored data recipe.
• A fitted cost-scaling law estimates that roughly $4,400 is sufficient to match Qwen2-1.5B under the authors’ evaluation protocol.
• The data, code, model weights, and complete training recipe are released under Apache 2.0.