EarlyEval: Cheaper Agent Evaluation via Early Outcome Prediction

TL;DR - EarlyEval reduces LLM-agent evaluation costs by predicting success or failure from intermediate behavior and stopping runs early. Across three agent benchmarks, it substantially cuts steps and token usage while largely preserving measured resolve rates.

• Uses paired LightGBM classifiers trained on behavioral, textual, and reference-solution features.
• Stops execution when either the success or failure classifier crosses a calibrated confidence threshold.
• Eliminates 13%–26% of agent steps, up to 44.1% of input tokens, and up to 29.4% of output tokens.
• Achieves 89%–97% prediction accuracy, with average per-agent resolve rates changing by only one to two percentage points.