When Does On-Policy Interaction Help? Representational Tradeoffs in Value-Based Imitation Learning
Ranking
Observed public metrics from 1 member.
Merged summary
TL;DR - A theoretical and empirical study showing that interactive on-policy expert querying in imitation learning lowers the learner's representational burden — it need only realize the expert's value function, not the expert's full policy. This matters for distillation settings where a smaller student cannot represent the teacher's policy.
- Introduces OVI, an interactive on-policy value-based IL algorithm that is statistically efficient under expert-value realizability and computationally efficient given a linear maximization oracle.
- Complementary negative result: without assumptions beyond expert-value realizability, any offline IL algorithm must scale with the complexity of the expert policy class — so interaction is necessary, not just convenient.
- Empirically, OVI beats offline policy-based (BC), interactive policy-based (DAgger), and offline value-based baselines, with the largest gains when the learner network is much less expressive than the expert.
- Frames the two known empirical tricks — expert interaction and value-function estimation — as interacting mechanisms rather than independent improvements.
Sources (1)
When Does On-Policy Interaction Help? Representational Tradeoffs in Value-Based Imitation Learning
TL;DR - A theoretical and empirical study showing that interactive on-policy expert querying in imitation learning lowers the learner's representational burden — it need only realize the expert's value function, not the expert's full policy. This matters for distillation settings where a smaller student cannot represent the teacher's policy.
- Introduces OVI, an interactive on-policy value-based IL algorithm that is statistically efficient under expert-value realizability and computationally efficient given a linear maximization oracle.
- Complementary negative result: without assumptions beyond expert-value realizability, any offline IL algorithm must scale with the complexity of the expert policy class — so interaction is necessary, not just convenient.
- Empirically, OVI beats offline policy-based (BC), interactive policy-based (DAgger), and offline value-based baselines, with the largest gains when the learner network is much less expressive than the expert.
- Frames the two known empirical tricks — expert interaction and value-function estimation — as interacting mechanisms rather than independent improvements.