HERALD: Counterfactual Audits and Minimal Repairs for Proof-of-Retrieval Rewards
Ranking
Observed public metrics from 1 member.
Merged summary
TL;DR - HERALD is an offline audit framework that stress-tests search-agent reward functions with exact same-question counterfactual interventions, revealing that a seemingly robust reward still falls to a label-free "citation-laundering" attack. It matters because high composite reward scores can mask ungrounded citations, so reward design for retrieval-augmented agents needs verifiable auditing before policy optimization.
- The audit separates candidate-visible from oracle information and enumerates detector contracts up front; on four Qwen3-8B pools over HotpotQA, 2WikiMultiHopQA, and MuSiQue, the baseline reward $R_0$ rejects search deletion and fake IDs but not laundering.
- A full $2^3$ ablation isolates term $L$ (citing a corpus passage absent from retrieved evidence) as the inclusion-minimal repair: $R[L]$ shows zero empirical attack success rate with a 0.50% one-sided cluster upper bound, holding across pool rules, a visible BM25 attacker, and four models.
- Under 5M-token matched training on 256 paired questions per benchmark, $R[L]$ passes the EM non-inferiority gate on HotpotQA and 2Wiki but not MuSiQue; citation precision and support recall rise 2.02 and 1.46 points and unsupported citations drop 1.69.
- Learning signal is extremely sparse — the detector fires in only 18 of 58,368 training trajectories — and broader hardening remains vulnerable when the attack strips an oracle support-ID penalty, separating robust scoring from actual policy transfer.
Sources (1)
HERALD: Counterfactual Audits and Minimal Repairs for Proof-of-Retrieval Rewards
TL;DR - HERALD is an offline audit framework that stress-tests search-agent reward functions with exact same-question counterfactual interventions, revealing that a seemingly robust reward still falls to a label-free "citation-laundering" attack. It matters because high composite reward scores can mask ungrounded citations, so reward design for retrieval-augmented agents needs verifiable auditing before policy optimization.
- The audit separates candidate-visible from oracle information and enumerates detector contracts up front; on four Qwen3-8B pools over HotpotQA, 2WikiMultiHopQA, and MuSiQue, the baseline reward $R_0$ rejects search deletion and fake IDs but not laundering.
- A full $2^3$ ablation isolates term $L$ (citing a corpus passage absent from retrieved evidence) as the inclusion-minimal repair: $R[L]$ shows zero empirical attack success rate with a 0.50% one-sided cluster upper bound, holding across pool rules, a visible BM25 attacker, and four models.
- Under 5M-token matched training on 256 paired questions per benchmark, $R[L]$ passes the EM non-inferiority gate on HotpotQA and 2Wiki but not MuSiQue; citation precision and support recall rise 2.02 and 1.46 points and unsupported citations drop 1.69.
- Learning signal is extremely sparse — the detector fires in only 18 of 58,368 training trajectories — and broader hardening remains vulnerable when the attack strips an oracle support-ID penalty, separating robust scoring from actual policy transfer.