ICML 2026 | 会查表却不会预测未来：南大TopBench测出大模型数据盲区

TL;DR - TopBench (Nanjing University, ICML 2026) is a benchmark for "implicit predictive reasoning" in tabular QA, where the answer isn't in the table and the model must first infer that a natural-language request is actually a prediction task, then build a model to answer it. It matters because current LLMs handle lookup/aggregation well but frequently fail to even enter prediction mode, a prerequisite for autonomous data-analysis agents.

• Setup: 35 real tables (Kaggle-sourced; healthcare, finance, daily consulting; <1K to >6M rows) and 779 queries across four task types — Single-Point Prediction (274), Decision Making (186), Treatment Effect Analysis (105), Ranking & Filtering (214); 384 regression vs 395 classification targets. Queries use dual personas (lay user vs data owner) and logic-driven sampling with hard-negative pairs and noisy candidate pools.
• Evaluation: LLM-as-a-Judge with string/fuzzy/NLI re-anchoring to the model's original text to curb judge hallucination; regression scored via a composite of point error, interval coverage and an over-wide-interval penalty; Ranking & Filtering scored deterministically on generated CSVs via F1, Set Recall, NDCG and batch NMAE.
• Results: Most models score below 0.60; the best reaches only ~0.65 on basic single-point prediction. Code execution isn't a fix — GPT-5.2 gains on Treatment Effect (0.51→0.65) but Qwen3-Instruct drops on single-point (0.57→0.43) by writing pandas filters instead of training a predictor, and Qwen3-Thinking loops through rows searching for an exact match ("Exhaustive Retrieval Loop") until context is exhausted.
• Two bottlenecks isolated: supplying target column/task type/feature descriptions lifts scores (Qwen3-Instruct 0.43→0.56; DeepSeek-V3.2 0.57→0.68), showing intent alignment is one blocker; a predict-only ensemble given gold task structure still beats the best agentic E2E run (0.66→0.76 single-point), showing tabular modeling quality is a second, independent blocker.