Benchmarking Multimodal Large Language Models for Scientific Visualization Literacy

TL;DR - A benchmark study evaluating whether multimodal LLMs can understand scientific visualizations (SciVis), using a standardized 49-item literacy test and comparing against 485 human participants. It matters because it exposes a gap in current MLLM evaluations, which are largely chart-centric and overlook scientific visualization competence.

• Six MLLMs (three closed-source, three open-source) were tested under a closed-world protocol on 49 items spanning 18 visualizations, 8 techniques, and 11 task types.
• Gemini was the strongest model, exceeding the human mean on evaluated subsets, while all open-source models stayed below the human baseline; SciVis literacy was uneven across models.
• Models did best on scientific illustration, search, and spatial understanding, but struggled with texture-based and integration-based visualizations and quantitative estimation.
• Recurring failures included fine-grained quantitative estimation, flow-direction interpretation, and grounded encoding interpretation; code and outputs are publicly released.