We present 360CityArena, a benchmark for evaluating the urban exploration capabilities of Embodied Agents within a photorealistic environment constructed from 360° videos. Existing outdoor benchmarks either lack sufficient photorealism and complexity, resulting in a considerable gap from real-world urban environments. 360CityArena is built on a realistic reconstruction of the Akihabara district in Tokyo, Japan, using 602 360° video segments covering 85 streets, and consists of 175 meticulously human-crafted tasks. It encompasses three task categories — Environment Understanding, Path Reasoning, and Spatial Reasoning — covering fundamental abilities required for urban exploration, such as localization, landmark search, path planning, and relational spatial reasoning, thereby enabling comprehensive evaluation in realistic urban scenes. Our evaluation using state-of-the-art LMM-based agents shows that even the strongest model, Gemini 2.5 Flash, performs far below human level (human: 77.3% vs. Gemini 2.5 Flash: 17.1%), revealing substantial challenges that remain in city-scale embodied navigation and reasoning. 360CityArena provides a necessary and challenging testbed for photorealistic urban-district navigation and spatial reasoning.
The environment used in 360CityArena is a Realistic Virtual World of Akihabara constructed from 602 360° video segments over 85 streets. The videos are projected onto a spherical surface and organized into a navigable pose graph in Unity, enabling agents to traverse the captured area. Agents therefore navigate along prerecorded 360° video trajectories represented as a pose graph, rather than moving freely to arbitrary 3D positions or physically interacting with the environment. The resulting pose graph forms a single connected component with 193 nodes and 305 edges, and has a mean branching degree of 3.16.
In terms of geographic coverage, 360CityArena spans approximately 750 m north–south and 650 m east–west around Akihabara Station. The area features diverse streetscapes with narrow sidewalks, complex building layouts, abundant outdoor ads, and electronic signboards, creating a visually dense, information-rich scene. Each street is captured twice (one per direction), reflecting direction-dependent visual changes.
Our benchmark consists of three major categories and seven subcategories. Each subcategory contains 25 tasks. Each task is labeled Easy, Medium, or Hard by annotators based on distance, instruction ambiguity, landmark visibility, and required exploration.
This category evaluates an agent’s ability to perceive and understand its surroundings, covering visual perception, memory, and language-grounded scene understanding.
This category evaluates an agent’s ability to plan and explore based on spatial understanding and to execute those plans in accordance with the spatial layout of the environment.
This category evaluates an agent’s ability to understand the structural layout of objects and the positional relationships between landmarks in the environment.
Examples in each task type in 360CityArena. Environment Understanding includes Localization and Landmark Search with Language or Image; Path Reasoning includes Map Navigation and Vision-Language Navigation; Spatial Reasoning includes Object Count and Relational Spatial Reasoning.
As shown in Table 2, human participants achieved higher accuracy than current LMMs. All LMMs still fall far short of human performance.
Human
77.3%
Gemini 2.5 Flash
17.1%
Our evaluation using state-of-the-art LMM-based agents shows that even the strongest model, Gemini 2.5 Flash, performs far below human level.
| Agent | Environment Understanding | Path Reasoning | Spatial Reasoning | Average | ||||
|---|---|---|---|---|---|---|---|---|
| Loc. | Landmark (Lang.) |
Landmark (Img.) |
Map Nav. | VLN | Obj. Count | Rel. Reason. |
||
| GPT-5 | 8.0 | 16.0 | 48.0 | 0.0 | 8.0 | 2.4 | 32.0 | 16.3 |
| Claude Sonnet 4.5 | 4.0 | 4.0 | 16.0 | 4.0 | 4.0 | 10.8 | 8.0 | 7.3 |
| Gemini 2.5 Flash | 12.0 | 28.0 | 36.0 | 0.0 | 8.0 | 24.0 | 12.0 | 17.1 |
| Qwen2.5-VL-32B-Instruct | 4.0 | 16.0 | 20.0 | 0.0 | 0.0 | 18.8 | 4.0 | 9.0 |
| InternVL3.5-8B | 4.0 | 20.0 | 20.0 | 0.0 | 12.0 | 2.8 | 0.0 | 8.4 |
| InternVL3.5-38B | 0.0 | 16.0 | 0.0 | 0.0 | 12.0 | 7.2 | 4.0 | 5.6 |
| Human | 68.0 | 64.0 | 92.0 | 92.0 | 88.0 | 45.2 | 92.0 | 77.3 |
Table 2: Overall Results (%). Comparison of model and human performance across seven spatial and reasoning tasks, grouped into three major categories. Gemini 2.5 Flash achieves the highest overall performance, while the strongest model varies across individual tasks. All LMMs still fall far short of human performance.
All LMMs fall far short of human performance across the benchmark. Gemini 2.5 Flash achieves the highest overall performance among the evaluated models, while the strongest model varies across individual tasks.
Across models, Landmark Search with Image generally outperforms Landmark Search with Text (e.g., GPT-5: 48.0 vs. 16.0; Claude: 16.0 vs. 4.0; Qwen: 20.0 vs. 16.0), suggesting that visual inputs provide concrete cues.
Model performance generally declines with higher task difficulty. GPT-5’s accuracy in Environment Understanding drops from 28.0 to 25.0 to 18.5, and in Path Reasoning from 11.1 to 0.0 to 0.0 across the Easy, Medium, and Hard settings.
@inproceedings{watanabe2026360cityarena,
title = {360CityArena: A Realistic Virtual Urban Navigation Benchmark for Embodied Agents},
author = {Watanabe, Kenta and Miyai, Atsuyuki and Takenawa, Mizuki and Aizawa, Kiyoharu and Yamasaki, Toshihiko},
booktitle = {Proceedings of the European Conference on Computer Vision (ECCV)},
year = {2026}
}@article{takenawa2026building,
title = {Building and Evaluating a Realistic Virtual World for Large Scale Urban Exploration from 360° Videos},
author = {Takenawa, Mizuki and Sugimoto, Naoki and W{\"o}hler, Leslie and Ikehata, Satoshi and Aizawa, Kiyoharu},
journal = {Multimedia Tools and Applications},
volume = {85},
number = {2},
pages = {149},
year = {2026}
}