When Does On-Policy Interaction Help? Representational Tradeoffs in Value-Based Imitation Learning

TL;DR - A theoretical and empirical study showing that interactive on-policy expert querying in imitation learning lowers the learner's representational burden — it need only realize the expert's value function, not the expert's full policy. This matters for distillation settings where a smaller student cannot represent the teacher's policy.

• Introduces OVI, an interactive on-policy value-based IL algorithm that is statistically efficient under expert-value realizability and computationally efficient given a linear maximization oracle.
• Complementary negative result: without assumptions beyond expert-value realizability, any offline IL algorithm must scale with the complexity of the expert policy class — so interaction is necessary, not just convenient.
• Empirically, OVI beats offline policy-based (BC), interactive policy-based (DAgger), and offline value-based baselines, with the largest gains when the learner network is much less expressive than the expert.
• Frames the two known empirical tricks — expert interaction and value-function estimation — as interacting mechanisms rather than independent improvements.