DeepStress: Stress-Testing Deep Search Agents
Merged summary
TL;DR — DeepStress is a stress-testing framework that probes how robust deep search agents are when fed poor-quality evidence, revealing failure modes that realistic benchmarks rarely expose. It matters because real-world retrieval-augmented agents routinely encounter unreliable documents.
- Replaces an agent's retrieval module with a controlled synthetic environment to tune the frequency of challenging evidence.
- Varies three document-reliability dimensions: trustworthiness, relevance, and factuality.
- Evaluated on HotpotQA and BrowseCompPlus, finding agents differ substantially in handling unreliable information.
- Proposes new metrics capturing system outcomes and the interplay between parametric (internal) and retrieved knowledge.
Sources (1)
DeepStress: Stress-Testing Deep Search Agents
TL;DR — DeepStress is a stress-testing framework that probes how robust deep search agents are when fed poor-quality evidence, revealing failure modes that realistic benchmarks rarely expose. It matters because real-world retrieval-augmented agents routinely encounter unreliable documents.
- Replaces an agent's retrieval module with a controlled synthetic environment to tune the frequency of challenging evidence.
- Varies three document-reliability dimensions: trustworthiness, relevance, and factuality.
- Evaluated on HotpotQA and BrowseCompPlus, finding agents differ substantially in handling unreliable information.
- Proposes new metrics capturing system outcomes and the interplay between parametric (internal) and retrieved knowledge.