Blind Men and the Elephant: Probing the Epistemic Myopia of LLMs under Long-Tail Divergent Knowledge
Ranking
Overall
85
Content
95
Popularity
63
Observed public metrics from 1 member.
Merged summary
TL;DR - ElephantBench evaluates whether LLMs retain multiple conflicting accounts of long-tail facts rather than only a dominant answer. Across 32 models, even the strongest recovered both accounts for just 52.4% of questions, revealing persistent incompleteness in parametric memory.
- The benchmark contains 1,094 closed-book QA questions built from naturally divergent accounts found through an auditable, graph-based pipeline.
- Answers are traceable to source documents, checked against authoritative public sources, and reviewed by human annotators.
- Larger models and inference-time reasoning improve recall but do not eliminate the tendency to omit one account.
- More balanced corpus exposure correlates with more complete recall, while exposure imbalance favors the dominant account.
Sources (1)
Blind Men and the Elephant: Probing the Epistemic Myopia of LLMs under Long-Tail Divergent Knowledge
Public signals
Hugging Face upvotes 19
TL;DR - ElephantBench evaluates whether LLMs retain multiple conflicting accounts of long-tail facts rather than only a dominant answer. Across 32 models, even the strongest recovered both accounts for just 52.4% of questions, revealing persistent incompleteness in parametric memory.
- The benchmark contains 1,094 closed-book QA questions built from naturally divergent accounts found through an auditable, graph-based pipeline.
- Answers are traceable to source documents, checked against authoritative public sources, and reviewed by human annotators.
- Larger models and inference-time reasoning improve recall but do not eliminate the tendency to omit one account.
- More balanced corpus exposure correlates with more complete recall, while exposure imbalance favors the dominant account.