Real-Time Detection and Repair of LLM Agent Failures
Ranking
Observed public metrics from 1 member.
Merged summary
TL;DR - An arXiv preprint showing that mid-episode LLM agent failures (loops, tool-error cascades, goal drift, fabrication) can be caught from cheap step telemetry plus deterministic verification, then repaired by rollback-and-rerun — at ~200 microseconds per step instead of an LLM-judge call. It matters because it makes continuous agent reliability monitoring affordable enough to run in production.
- A one-class echo-state-network ensemble with CUSUM alarms, trained only on healthy runs, detects 0.71 of failures at a 5% false-alarm budget (AUROC 0.872) over 2,823 episodes spanning three frameworks, three local models (qwen2.5 7b/3b, llama3.1 8b) and gemini-2.5-flash; its edge over a memoryless baseline grows with post-onset horizon (+0.09 at ≤3 steps, +0.40 at ≥9).
- Ranking transfers without retraining to external corpora (AFTraj-2K 0.745, ATBench 0.779), but the healthy null does not transfer: AUROC 0.527 cold vs 0.885 after recalibration, so each deployment needs its own baseline.
- A deterministic verification layer — recomputing the run's stated total from actual tool results and checking every required call was made — catches 60% of failures (96% with the coverage check) at 0/63 false positives, beats the monitor's 54% at 17% FPR, transfers unchanged to llama3.1:8b (110/110 at 0/10), and fires on 0 of 1825 healthy episodes.
- Closing detection into repair by rolling back and re-running flagged episodes recovers 45% of failures vs a 16% resampling control (p=0.0005), raising task success from 52% to 73% for roughly one extra model call per run; code, traces and results are released.
Sources (1)
Real-Time Detection and Repair of LLM Agent Failures
TL;DR - An arXiv preprint showing that mid-episode LLM agent failures (loops, tool-error cascades, goal drift, fabrication) can be caught from cheap step telemetry plus deterministic verification, then repaired by rollback-and-rerun — at ~200 microseconds per step instead of an LLM-judge call. It matters because it makes continuous agent reliability monitoring affordable enough to run in production.
- A one-class echo-state-network ensemble with CUSUM alarms, trained only on healthy runs, detects 0.71 of failures at a 5% false-alarm budget (AUROC 0.872) over 2,823 episodes spanning three frameworks, three local models (qwen2.5 7b/3b, llama3.1 8b) and gemini-2.5-flash; its edge over a memoryless baseline grows with post-onset horizon (+0.09 at ≤3 steps, +0.40 at ≥9).
- Ranking transfers without retraining to external corpora (AFTraj-2K 0.745, ATBench 0.779), but the healthy null does not transfer: AUROC 0.527 cold vs 0.885 after recalibration, so each deployment needs its own baseline.
- A deterministic verification layer — recomputing the run's stated total from actual tool results and checking every required call was made — catches 60% of failures (96% with the coverage check) at 0/63 false positives, beats the monitor's 54% at 17% FPR, transfers unchanged to llama3.1:8b (110/110 at 0/10), and fires on 0 of 1825 healthy episodes.
- Closing detection into repair by rolling back and re-running flagged episodes recovers 45% of failures vs a 16% resampling control (p=0.0005), raising task success from 52% to 73% for roughly one extra model call per run; code, traces and results are released.