Why Sample What You Can Enumerate? Exact Policy Optimization for Genomic Tool Selection

TL;DR - FGPO exactly enumerates and scores all tool subsets when training frozen genomic reasoners, avoiding GRPO’s increasingly weak reward signal. It outperformed GRPO across 15 benchmark settings while reducing reward-evaluation costs and unnecessary tool use.

• GRPO’s sampled, group-normalized advantages can vanish as the policy repeatedly selects preferred subsets; genomic questions with no reward signal rose from 0.2% to 20.8% after training.
• FGPO optimizes the exact expectation over the complete action space and precomputes rewards for every question–subset pair.
• Across five frozen reasoners and three genomic benchmarks, FGPO beat GRPO in all 15 settings by 6.75 points on average and up to 14.20 points.
• Compared with on-demand GRPO, FGPO avoided a schedule requiring 2.4× more reward evaluations and reduced GenomeQA tool calls from 2.36 to 1.40 per question.