Do Agent Optimizers Compound? A Continual-Learning Evaluation on Terminal-Bench 2.0
Ranking
Observed public metrics from 1 member.
Merged summary
TL;DR — A continual-learning evaluation on Terminal-Bench 2.0 tests whether agent-optimization gains compound across successive optimization rounds, finding that only regression-controlled optimization sustains and grows improvements. This matters because deployed agents are optimized recursively as new tasks and failures appear, not just once against a fixed benchmark.
- Compares three agent-harness optimizers (GEPA, Meta Harness, RELAI-VCL) under identical budgets in a two-phase continual setup; all three beat the baseline in the conventional single-phase static test.
- Behaviors diverge once new tasks arrive: GEPA transfers below the unoptimized baseline, Meta Harness transfers well but stops improving with a second budget, and RELAI-VCL both transfers positively and keeps improving.
- RELAI-VCL reaches the highest lifelong average pass rate (76.4% vs. 66.0% GEPA, 64.6% Meta Harness, 58.7% baseline).
- Key claim: gains compound only when regression control is built into the optimization loop, biasing against non-generalizing shortcut solutions.
Sources (1)
Do Agent Optimizers Compound? A Continual-Learning Evaluation on Terminal-Bench 2.0
TL;DR — A continual-learning evaluation on Terminal-Bench 2.0 tests whether agent-optimization gains compound across successive optimization rounds, finding that only regression-controlled optimization sustains and grows improvements. This matters because deployed agents are optimized recursively as new tasks and failures appear, not just once against a fixed benchmark.
- Compares three agent-harness optimizers (GEPA, Meta Harness, RELAI-VCL) under identical budgets in a two-phase continual setup; all three beat the baseline in the conventional single-phase static test.
- Behaviors diverge once new tasks arrive: GEPA transfers below the unoptimized baseline, Meta Harness transfers well but stops improving with a second budget, and RELAI-VCL both transfers positively and keeps improving.
- RELAI-VCL reaches the highest lifelong average pass rate (76.4% vs. 66.0% GEPA, 64.6% Meta Harness, 58.7% baseline).
- Key claim: gains compound only when regression control is built into the optimization loop, biasing against non-generalizing shortcut solutions.