Can We Trust Item Response Theory for AI Evaluation?
Merged summary
TL;DR - A simulation study interrogating whether item response theory (IRT), increasingly used to estimate LLM capabilities and rank models on benchmarks, remains reliable when applied to AI evaluation data that violates the assumptions IRT was built for. It matters because flawed IRT inference could distort widely-cited benchmark rankings and quality claims.
- AI benchmark data differs from human-testing regimes IRT was designed for: few models, many items, and skewed/clustered/multimodal capability distributions.
- Using parameters from six widely used LLM benchmarks, the authors simulated response matrices under three IRT models across ~18,000 conditions, comparing four estimators: marginal maximum likelihood, MCMC, variational inference, and a neural pseudo-Siamese estimator.
- Findings: classical estimators can become computationally infeasible at benchmark scale, while scalable estimators yield unreliable item-level and ranking inferences with small or non-normal model sets.
- The paper offers guidance on required sample sizes and diagnostics for trustworthy IRT use, flagging when latent-trait models support versus distort benchmarking claims.
Sources (1)
Can We Trust Item Response Theory for AI Evaluation?
TL;DR - A simulation study interrogating whether item response theory (IRT), increasingly used to estimate LLM capabilities and rank models on benchmarks, remains reliable when applied to AI evaluation data that violates the assumptions IRT was built for. It matters because flawed IRT inference could distort widely-cited benchmark rankings and quality claims.
- AI benchmark data differs from human-testing regimes IRT was designed for: few models, many items, and skewed/clustered/multimodal capability distributions.
- Using parameters from six widely used LLM benchmarks, the authors simulated response matrices under three IRT models across ~18,000 conditions, comparing four estimators: marginal maximum likelihood, MCMC, variational inference, and a neural pseudo-Siamese estimator.
- Findings: classical estimators can become computationally infeasible at benchmark scale, while scalable estimators yield unreliable item-level and ranking inferences with small or non-normal model sets.
- The paper offers guidance on required sample sizes and diagnostics for trustworthy IRT use, flagging when latent-trait models support versus distort benchmarking claims.