LittleLearner: Language Models Under Pedagogically Controlled Knowledge Exposure

TL;DR - LittleLearner is a 5B-parameter language model trained from scratch on an 88B-token corpus restricted to U.S. elementary-school material. It provides a controlled sandbox for studying knowledge acquisition with interpretable exposure and capability boundaries.

• LITTLECURRICULUM excludes concepts, facts, and vocabulary taught above Grade 5.
• The resulting model retains enough language competence for open-ended evaluation while exhibiting curriculum-aligned knowledge limits.
• Post-training and in-context learning improved the use of existing knowledge but did not increase out-of-scope capabilities.
• The authors release both the corpus and model for research on how models acquire, represent, and apply knowledge.