The Low Frequency Trap: Video Language Models Fail at Simple Event Bookkeeping
Ranking
Observed public metrics from 1 member.
Merged summary
TL;DR - An arXiv study introduces "trace-grounded parametric profiling," a controlled video benchmark that audits which events a video-language model reports against executable ground-truth traces, not just the final count. It shows current VLMs collapse at simple event bookkeeping as event count and frequency rise, exposing temporal reasoning failures that aggregate accuracy hides.
- Benchmark spans 2,190 synthetic videos across three tasks (bouncing-ball wall contacts, visual blinks, categorical state transitions), varying event count N and frequency F while holding rendering fixed, with per-video executable event traces enabling timestamp-level evaluation.
- Failure is staged and representation-dependent: at an 80% reliability threshold, Gemini 3.6 Flash counts persistent state transitions up to 12 events at 0.5–1.0 Hz, but has no reliable positive-count region for transient blinks.
- In the high-count/high-frequency regime only 0.2% of final counts are correct and just 18.1% of true events are recovered; higher sampling raises Bounce Ball accuracy 19.6% → 29.3% yet the reported sequence matches ground truth only 3.7% of the time.
- Extra frames and alternative prompting strategies inflate scores without faithful event recovery; real-world video evaluations show the same concentration of success at low event counts.
Sources (1)
The Low Frequency Trap: Video Language Models Fail at Simple Event Bookkeeping
TL;DR - An arXiv study introduces "trace-grounded parametric profiling," a controlled video benchmark that audits which events a video-language model reports against executable ground-truth traces, not just the final count. It shows current VLMs collapse at simple event bookkeeping as event count and frequency rise, exposing temporal reasoning failures that aggregate accuracy hides.
- Benchmark spans 2,190 synthetic videos across three tasks (bouncing-ball wall contacts, visual blinks, categorical state transitions), varying event count N and frequency F while holding rendering fixed, with per-video executable event traces enabling timestamp-level evaluation.
- Failure is staged and representation-dependent: at an 80% reliability threshold, Gemini 3.6 Flash counts persistent state transitions up to 12 events at 0.5–1.0 Hz, but has no reliable positive-count region for transient blinks.
- In the high-count/high-frequency regime only 0.2% of final counts are correct and just 18.1% of true events are recovered; higher sampling raises Bounce Ball accuracy 19.6% → 29.3% yet the reported sequence matches ground truth only 3.7% of the time.
- Extra frames and alternative prompting strategies inflate scores without faithful event recovery; real-world video evaluations show the same concentration of success at low event counts.