「用初中数学讲明白AI」第6章：从"自说自话"到"有问必答"——微调与对齐

TL;DR - An educational explainer describes how instruction tuning, RLHF, and DPO turn pretrained language models into useful assistants aligned with human preferences. It argues that post-training quality—not parameter count alone—was central to ChatGPT’s usability and adoption.

• Instruction tuning uses curated question-answer examples to shift models from text completion toward following user requests.
• RLHF trains a reward model from ranked responses, then optimizes the LLM against that learned preference signal.
• DPO learns directly from preferred and rejected responses, avoiding RLHF’s separate reward-model and reinforcement-learning stages.
• Human-feedback alignment can introduce reward hacking, evaluator bias, and difficulty judging outputs that exceed annotators’ expertise.