Why Large Language Models Fail at Tabular Prediction

TL;DR - An arXiv study isolates why frontier LLMs underperform classical models on tabular prediction, finding that input dimensionality — not noise, formatting, tokenization, or batching — is the decisive failure factor. It matters because it grounds the case for dedicated tabular foundation models rather than generic LLM prompting.

• Tested a frontier LLM in a pure inference regime: one generation pass over a prompt holding full train + test data, with no tools, agentic scaffolding, or fine-tuning.
• Controlled experiments falsified four hypotheses: noisy/non-linearly-separable data, linearised CSV obscuring column structure, numeric tokenization, and number of test points per query.
• Sweeping random linear projections across 31 benchmark datasets, the LLM was the only one of nine methods whose accuracy decreased with rising dimensionality; all classical baselines stayed flat or improved.
• Behaviourally, the LLM mimics a local distance-based method in 2D (up to 91.6% grid agreement), but in higher dimensions none of 252 configured classical models — even with tuned dimension-dependent noise — reproduces its predictions; the authors explicitly leave the internal mechanism open.