EarthVerse: Benchmarking Scientific Agents Across Dynamic Earth Systems and Natural Hazards
Ranking
Overall
88
Content
100
Popularity
59
Observed public metrics from 1 member.
Merged summary
TL;DR - EarthVerse is a benchmark for evaluating tool-using scientific agents on 405 reproducible Earth-system and natural-hazard investigations. Results show that even strong systems struggle to maintain reliable end-to-end reasoning across heterogeneous evidence, calculations, units, and physical interpretation.
- Tasks span 199 documented events and 19 hazard families, with provenance-preserving answers and executable ground truth.
- The benchmark evaluates evidence selection, tool use, memory, reasoning, interaction, and scientific execution under a controlled protocol.
- Across 25 model and agent systems, the best mean answer-unit accuracy was 84.65%, but the highest Strict@95 score was only 34.81%.
- This gap indicates that agents often solve individual steps correctly while failing to preserve a consistent scientific chain across the full investigation.
Sources (1)
EarthVerse: Benchmarking Scientific Agents Across Dynamic Earth Systems and Natural Hazards
Public signals
Hugging Face upvotes 1
TL;DR - EarthVerse is a benchmark for evaluating tool-using scientific agents on 405 reproducible Earth-system and natural-hazard investigations. Results show that even strong systems struggle to maintain reliable end-to-end reasoning across heterogeneous evidence, calculations, units, and physical interpretation.
- Tasks span 199 documented events and 19 hazard families, with provenance-preserving answers and executable ground truth.
- The benchmark evaluates evidence selection, tool use, memory, reasoning, interaction, and scientific execution under a controlled protocol.
- Across 25 model and agent systems, the best mean answer-unit accuracy was 84.65%, but the highest Strict@95 score was only 34.81%.
- This gap indicates that agents often solve individual steps correctly while failing to preserve a consistent scientific chain across the full investigation.