AgentHPOBench: A Benchmark For Evaluating LLM Agents as Sequential Hyperparameter Optimizers

TL;DR - AgentHPOBench is a benchmark that tests whether LLM agents can act as sequential hyperparameter optimizers — reading experimental logs and metrics to propose the next configuration — rather than just generating static code. It matters because it targets the iterative, evidence-driven loop that "autonomous scientific agent" claims depend on.

• 30 executable ML tasks spanning seven research categories; each starts from a validated baseline run followed by several sequential agent interventions.
• At every step the agent sees accumulated configurations, metrics, and logs, then must propose the next valid configuration — closing the gap left by static code-generation and final-answer benchmarks.
• 12 widely used agents plus conventional HPO baselines were evaluated under a unified protocol.
• Agents show measurable optimization ability across domains but falter at sustained iterative refinement, complex log diagnosis, and reaching reported reference performance.