Benchmarking Multimodal Large Language Models for Scientific Visualization Literacy
Merged summary
TL;DR - A benchmark study evaluating whether multimodal LLMs can understand scientific visualizations (SciVis), using a standardized 49-item literacy test and comparing against 485 human participants. It matters because it exposes a gap in current MLLM evaluations, which are largely chart-centric and overlook scientific visualization competence.
- Six MLLMs (three closed-source, three open-source) were tested under a closed-world protocol on 49 items spanning 18 visualizations, 8 techniques, and 11 task types.
- Gemini was the strongest model, exceeding the human mean on evaluated subsets, while all open-source models stayed below the human baseline; SciVis literacy was uneven across models.
- Models did best on scientific illustration, search, and spatial understanding, but struggled with texture-based and integration-based visualizations and quantitative estimation.
- Recurring failures included fine-grained quantitative estimation, flow-direction interpretation, and grounded encoding interpretation; code and outputs are publicly released.
Sources (1)
Benchmarking Multimodal Large Language Models for Scientific Visualization Literacy
TL;DR - A benchmark study evaluating whether multimodal LLMs can understand scientific visualizations (SciVis), using a standardized 49-item literacy test and comparing against 485 human participants. It matters because it exposes a gap in current MLLM evaluations, which are largely chart-centric and overlook scientific visualization competence.
- Six MLLMs (three closed-source, three open-source) were tested under a closed-world protocol on 49 items spanning 18 visualizations, 8 techniques, and 11 task types.
- Gemini was the strongest model, exceeding the human mean on evaluated subsets, while all open-source models stayed below the human baseline; SciVis literacy was uneven across models.
- Models did best on scientific illustration, search, and spatial understanding, but struggled with texture-based and integration-based visualizations and quantitative estimation.
- Recurring failures included fine-grained quantitative estimation, flow-direction interpretation, and grounded encoding interpretation; code and outputs are publicly released.