What Current AI Benchmarks Leave Unmeasured: Modality, Search, Citations, and Implications (for Safety Evaluations)
Ranking
Overall
64
Content
75
Popularity
39
Observed public metrics from 1 member.
Merged summary
TL;DR - An arXiv audit showing that standard LLM benchmark practice (single access modality, single run, accuracy-only reporting) hides substantial behavioral variation, so safety claims built on those numbers are shakier than they appear.
- Compared ChatGPT's chat UI vs. OpenAI's API, with and without web search, using 401 stratified prompts from BBQ and SafetyBench and 4,812 responses over three runs per prompt.
- Chat UI was less accurate than the API on both benchmarks with search disabled; enabling web search cut accuracy by up to 8 percentage points and even reversed the modality performance ordering on one benchmark.
- Repeated runs of the same prompt gave inconsistent responses for up to 21% of prompts, and the two modalities cited different sources and abstained inconsistently.
- Authors argue safety evaluations should systematically report modality, multi-run consistency, search conditions, and response-level behaviors (citation grounding, abstention) rather than accuracy alone.
Sources (1)
What Current AI Benchmarks Leave Unmeasured: Modality, Search, Citations, and Implications (for Safety Evaluations)
Public signals
Semantic Scholar citations 0 · Semantic Scholar influential citations 0
TL;DR - An arXiv audit showing that standard LLM benchmark practice (single access modality, single run, accuracy-only reporting) hides substantial behavioral variation, so safety claims built on those numbers are shakier than they appear.
- Compared ChatGPT's chat UI vs. OpenAI's API, with and without web search, using 401 stratified prompts from BBQ and SafetyBench and 4,812 responses over three runs per prompt.
- Chat UI was less accurate than the API on both benchmarks with search disabled; enabling web search cut accuracy by up to 8 percentage points and even reversed the modality performance ordering on one benchmark.
- Repeated runs of the same prompt gave inconsistent responses for up to 21% of prompts, and the two modalities cited different sources and abstained inconsistently.
- Authors argue safety evaluations should systematically report modality, multi-run consistency, search conditions, and response-level behaviors (citation grounding, abstention) rather than accuracy alone.