UrbanGround: From Local Perception to Spatial Agency in a Real-Scale City

TL;DR - UrbanGround is a closed-loop benchmark for testing whether multimodal LLM agents can turn street-level perception into reliable navigation in a physically constrained 3D replica of Hong Kong. Current agents handle basic visual recognition and short-range spatial reasoning, but struggle to sustain and correct goal-directed behavior over longer routes.

• Built from territory-wide 3D geospatial data, the sandbox supports first-person exploration and interactive-map navigation.
• Evaluates active spatial grounding, navigation to increasingly distant or ambiguous destinations, and robustness to route changes and pedestrian motion.
• Orientation and pedestrian-aware movement remain unreliable despite useful local perception capabilities.
• During extended exploration, errors accumulate because agents fail to compose local skills into sustained behavior or recover effectively.