Coupled Calibration and Learning: Mitigating Teacher Bias in LLM Distillation without Target-Domain Reward Feedback

TL;DR - Coupled Calibration and Learning (CCL) is an LLM distillation method that jointly calibrates a biased teacher and trains a student under target-domain covariate shift, without target-domain reward feedback. The authors theoretically show that CCL can recover the optimal student within its policy class, whereas direct teacher matching may retain persistent error.

• CCL uses source-domain reward feedback to calibrate the teacher, then trains the student on target questions through token-level branching.
• Teacher calibration and student learning form an iterative feedback loop, with each updated student informing subsequent calibration.
• In an autoregressive policy framework, the student’s expected average KL divergence to an oracle student converges to zero at a polynomial rate.
• Regularized direct matching can remain bounded away from the oracle even when the teacher outperforms every student policy on regularized target reward.