‹ back to 2026-07-24

ICAE-Bench: Evaluating Coding Agents as Interactive Project Builders

Research LLM Agents

Ranking

Overall 83

Content 90

Popularity 67

Observed public metrics from 1 member.

Merged summary

TL;DR - ICAE-Bench evaluates coding agents as interactive project builders, starting from ambiguous product requirements rather than fully specified tasks. It measures whether agents can clarify requirements, use tools, debug, and reconstruct working repository-level software.

Tasks derive controlled ambiguity from real open-source repositories with executable behavior.
An automated User Agent reveals grounded hidden constraints without adding requirements or leaking implementation details.
Standardized black-box tests assess functional correctness alongside semantic, API, and structural fidelity.
Diagnostics also evaluate design quality and the quality of agent-user interaction.

Sources (1)

ICAE-Bench: Evaluating Coding Agents as Interactive Project Builders

arXiv cs.AI Zhongyuan Peng, Dan Huang, Chuyu Zhang, Caijun Xu, Changyi Xiao, Shibo Hong, David Lo, Lin Qiu, Xuezhi Cao, Jiyuan He, Yixin Cao 2026-07-23 arXiv:2607.21217

Public signals Hugging Face upvotes 7

Providers: Hugging Face · Upvotes 7 OpenAlex · N/A Publisher · N/A Semantic Scholar · N/A X · N/A Fetched 2026-08-23 14:28:29.691709 UTC

Tasks derive controlled ambiguity from real open-source repositories with executable behavior.
An automated User Agent reveals grounded hidden constraints without adding requirements or leaking implementation details.
Standardized black-box tests assess functional correctness alongside semantic, API, and structural fidelity.
Diagnostics also evaluate design quality and the quality of agent-user interaction.

item →