Open Evaluation Agent: Efficient and Promptable Evaluation of Visual Generative Models
Ranking
Observed public metrics from 1 member.
Merged summary
TL;DR - An arXiv paper proposing Evaluation Agent, an LLM-agent framework that evaluates text-to-image/video generative models through few-sample, multi-round, human-like probing instead of exhaustive sampling, plus Open-EA, an open-weight version of the planner. It matters because it cuts evaluation cost to ~10% of traditional benchmarking while adding promptability and explanations.
- Given a natural-language evaluation request, the agent decomposes it into sub-aspects, generates targeted prompts, samples from the model under test, calls suitable evaluation tools, and iteratively revises its plan from observed evidence — covering both predefined benchmark dimensions and open-ended user concerns.
- Reported efficiency: comparable results at ~10% of the evaluation time of conventional pipelines; validated on established T2I/T2V benchmarks and open-ended queries.
- Open-EA is built from EA-CoT-10K, a corpus of history-conditioned, step-level instruction-tuning records distilled from multi-round evaluation rollouts; EA-3B is fine-tuned from Qwen2.5-3B-Instruct as a local planning backbone preserving structured reasoning, tool invocation, and summary protocol, reducing reliance on proprietary APIs.
- Open-EA was tested on four in-domain and three out-of-domain T2V generator families, showing only partial cross-family transfer of the learned policy — a stated generalization limitation.
Sources (1)
Open Evaluation Agent: Efficient and Promptable Evaluation of Visual Generative Models
TL;DR - An arXiv paper proposing Evaluation Agent, an LLM-agent framework that evaluates text-to-image/video generative models through few-sample, multi-round, human-like probing instead of exhaustive sampling, plus Open-EA, an open-weight version of the planner. It matters because it cuts evaluation cost to ~10% of traditional benchmarking while adding promptability and explanations.
- Given a natural-language evaluation request, the agent decomposes it into sub-aspects, generates targeted prompts, samples from the model under test, calls suitable evaluation tools, and iteratively revises its plan from observed evidence — covering both predefined benchmark dimensions and open-ended user concerns.
- Reported efficiency: comparable results at ~10% of the evaluation time of conventional pipelines; validated on established T2I/T2V benchmarks and open-ended queries.
- Open-EA is built from EA-CoT-10K, a corpus of history-conditioned, step-level instruction-tuning records distilled from multi-round evaluation rollouts; EA-3B is fine-tuned from Qwen2.5-3B-Instruct as a local planning backbone preserving structured reasoning, tool invocation, and summary protocol, reducing reliance on proprietary APIs.
- Open-EA was tested on four in-domain and three out-of-domain T2V generator families, showing only partial cross-family transfer of the learned policy — a stated generalization limitation.