Open Evaluation Agent: Efficient and Promptable Evaluation of Visual Generative Models

TL;DR - An arXiv paper proposing Evaluation Agent, an LLM-agent framework that evaluates text-to-image/video generative models through few-sample, multi-round, human-like probing instead of exhaustive sampling, plus Open-EA, an open-weight version of the planner. It matters because it cuts evaluation cost to ~10% of traditional benchmarking while adding promptability and explanations.

• Given a natural-language evaluation request, the agent decomposes it into sub-aspects, generates targeted prompts, samples from the model under test, calls suitable evaluation tools, and iteratively revises its plan from observed evidence — covering both predefined benchmark dimensions and open-ended user concerns.
• Reported efficiency: comparable results at ~10% of the evaluation time of conventional pipelines; validated on established T2I/T2V benchmarks and open-ended queries.
• Open-EA is built from EA-CoT-10K, a corpus of history-conditioned, step-level instruction-tuning records distilled from multi-round evaluation rollouts; EA-3B is fine-tuned from Qwen2.5-3B-Instruct as a local planning backbone preserving structured reasoning, tool invocation, and summary protocol, reducing reliance on proprietary APIs.
• Open-EA was tested on four in-domain and three out-of-domain T2V generator families, showing only partial cross-family transfer of the learned policy — a stated generalization limitation.