CreativeInstruct: Scalably Teaching LLMs to Balance Quality, Creativity, and Diversity

TL;DR - CreativeInstruct is an instruction-tuning method that teaches LLMs to emit special [StartCreativity] spans, letting a single post-trained model recover base-model-like creativity and diversity without losing output quality. It matters because post-training's diversity collapse hurts both creative writing and downstream RL exploration.

• Injects learned [StartCreativity] control spans during instruction tuning to bias generation toward creative modes, avoiding the need for multiple models at inference time.
• Introduces a structural diversity metric based on graph edit distance, aimed at capturing narrative-level variation that lexical and semantic metrics miss.
• On narrative generation it matches or exceeds diversity of multi-model baselines and their distilled variants at equal quality; human annotators rated its outputs more creative than post-trained LLM outputs in 70.3% of cases.
• Creativity transfers to RL: GRPO on a CreativeInstruct checkpoint gains ~4% on AMC and ~5 points on MATH versus the same training from the standard post-trained checkpoint.