What Current AI Benchmarks Leave Unmeasured: Modality, Search, Citations, and Implications (for Safety Evaluations)

TL;DR - An arXiv audit showing that standard LLM benchmark practice (single access modality, single run, accuracy-only reporting) hides substantial behavioral variation, so safety claims built on those numbers are shakier than they appear.

• Compared ChatGPT's chat UI vs. OpenAI's API, with and without web search, using 401 stratified prompts from BBQ and SafetyBench and 4,812 responses over three runs per prompt.
• Chat UI was less accurate than the API on both benchmarks with search disabled; enabling web search cut accuracy by up to 8 percentage points and even reversed the modality performance ordering on one benchmark.
• Repeated runs of the same prompt gave inconsistent responses for up to 21% of prompts, and the two modalities cited different sources and abstained inconsistently.
• Authors argue safety evaluations should systematically report modality, multi-run consistency, search conditions, and response-level behaviors (citation grounding, abstention) rather than accuracy alone.