Stable Curves, Unstable Items: Item-Level Scaling Heterogeneity in Video LLMs

TL;DR - An arXiv cs.CV study showing that smooth aggregate scaling curves for Video LLMs (accuracy vs. visual budget) hide large, opposing per-item swings, meaning no single frame/resolution budget is optimal for all items. It matters because standard budget-scaling evaluation can mask regressions and leaves substantial accuracy headroom on the table.

• Across five open Video LLMs (three architecture families), four MCQA splits, open-ended QA, summarization, and fixed-history dialogue, item-level oracle headroom spans 8.8–18.9 accuracy points on the four-model matched MCQA grid, and 12.5–25.5% of items are correct at a lower budget but wrong at a higher one.
• The same complementarity appears in continuous metrics: Token-F1 oracle gaps of 2.7–3.7 points on MLVU generation and 3.8–4.8 points on AVSD current-turn generation, even where mean quality rises with budget.
• Effects persist across frame count, spatial resolution, sampling policy, temporal–spatial allocation, and independent raw-video vs. cached pipelines; the authors define matched-grid measures of configuration complementarity, harmful transitions, and text overwrite.
• Practical upshots: a controlled sampling intervention recovers 29.0% of terminal regressions, and a confidence cascade matches fixed-128-frame accuracy while cutting average shared frame cost 31.7%; per-item trajectories, provenance, annotations, and analysis code are released as an auditing artifact.