CAFE: Self-Improving Search Agents Need Co-Evolving Feedback

TL;DR - CAFE jointly improves an agentic search policy and its feedback critic by training them in alternating, coupled roles. Across seven in-domain and six out-of-domain benchmarks, this approach outperformed evaluated RL-based search agents on average and reduced answer-level hallucinations.

• A shared-parameter model alternates between acting as the search agent and providing corrective, in-trajectory feedback.
• Online RL rewards useful feedback requests using a call-versus-skip success gap and reshapes token-level advantages around feedback.
• Offline preference optimization trains feedback from matched successful and unsuccessful rollout trajectories.
• Ablations found that improving only the agent or critic eventually plateaued, while alternating updates continued to improve performance.