Zero Gap Is Not Restoration: Stratified Per-Question Probability Evaluation and Step-wise Mitigation of Benchmark Contamination
Ranking
Observed public metrics from 1 member.
Merged summary
TL;DR - An arXiv paper arguing that the standard G-AP metric for judging benchmark-contamination mitigation is misleading, and proposing both a better metric (SA-PPG) and a decoding-time mitigation method (RailCap). It matters because it suggests prior claims of "restoring" a contaminated model's true capability are substantially overstated.
- G-AP critique: discrete correct/incorrect readouts hide per-question behavior, averaging before differencing lets over- and under-suppression cancel, and uniform per-question weighting can be gamed by shifting solve probabilities onto the clean model's high-frequency values.
- SA-PPG: estimates each question's solve probability by sampling, differences it against the clean model per question, then aggregates within strata defined by the clean model's solve probability.
- RailCap: judges contamination during generation rather than pre-estimating where it lies — when a sample falls back onto the greedy trajectory, the next trajectory token is capped to the runner-up, accumulating suppression until the response distribution is sufficiently dispersed.
- Findings: across multiple contaminated models and benchmarks, SA-PPG shows prior strategies' restoration is overestimated, and RailCap achieves the lowest SA-PPG (no numeric results given in the provided abstract).
Sources (1)
Zero Gap Is Not Restoration: Stratified Per-Question Probability Evaluation and Step-wise Mitigation of Benchmark Contamination
TL;DR - An arXiv paper arguing that the standard G-AP metric for judging benchmark-contamination mitigation is misleading, and proposing both a better metric (SA-PPG) and a decoding-time mitigation method (RailCap). It matters because it suggests prior claims of "restoring" a contaminated model's true capability are substantially overstated.
- G-AP critique: discrete correct/incorrect readouts hide per-question behavior, averaging before differencing lets over- and under-suppression cancel, and uniform per-question weighting can be gamed by shifting solve probabilities onto the clean model's high-frequency values.
- SA-PPG: estimates each question's solve probability by sampling, differences it against the clean model per question, then aggregates within strata defined by the clean model's solve probability.
- RailCap: judges contamination during generation rather than pre-estimating where it lies — when a sample falls back onto the greedy trajectory, the next trajectory token is capped to the runner-up, accumulating suppression until the response distribution is sufficiently dispersed.
- Findings: across multiple contaminated models and benchmarks, SA-PPG shows prior strategies' restoration is overestimated, and RailCap achieves the lowest SA-PPG (no numeric results given in the provided abstract).