We present 360CityArena, a benchmark for evaluating the urban exploration capabilities of Embodied Agents within a photorealistic environment constructed from 360° videos. Existing outdoor benchmarks lack sufficient photorealism or complexity, resulting in a considerable gap from real-world urban environments. 360CityArena is built on a realistic reconstruction of the Akihabara district in Tokyo, Japan, using 602 360° video segments covering 85 streets, and consists of 175 meticulously human-crafted tasks. It encompasses three task categories — Environment Understanding, Path Reasoning, and Spatial Reasoning — covering fundamental abilities required for urban exploration, such as localization, landmark search, path planning, and relational spatial reasoning, thereby enabling comprehensive evaluation in realistic urban scenes. Our evaluation of 14 LMM-based agents shows that the strongest evaluated model, GPT-6 Astra, achieves a mean score of 43.8%, compared with 77.3% for a human baseline of five participants familiar with Akihabara. This gap reveals substantial challenges that remain in city-scale embodied navigation and reasoning. 360CityArena provides a necessary and challenging testbed for photorealistic urban-district navigation and spatial reasoning.
The environment used in 360CityArena is a Realistic Virtual World of Akihabara constructed from 602 360° video segments over 85 streets. The videos are projected onto a spherical surface and organized into a navigable pose graph in Unity, enabling agents to traverse the captured area. Agents therefore navigate along prerecorded 360° video trajectories represented as a pose graph, rather than moving freely to arbitrary 3D positions or physically interacting with the environment. The resulting pose graph forms a single connected component with 193 nodes and 305 edges, and has a mean branching degree of 3.16.
In terms of geographic coverage, 360CityArena spans approximately 750 m north–south and 650 m east–west around Akihabara Station. The area features diverse streetscapes with narrow sidewalks, complex building layouts, abundant outdoor ads, and electronic signboards, creating a visually dense, information-rich scene. Each street is captured twice (one per direction), reflecting direction-dependent visual changes.
Our benchmark consists of three major categories and seven subcategories. Each subcategory contains 25 tasks. Each task is labeled Easy, Medium, or Hard by annotators based on distance, instruction ambiguity, landmark visibility, and required exploration.
This category evaluates an agent’s ability to perceive and understand its surroundings, covering visual perception, memory, and language-grounded scene understanding.
This category evaluates an agent’s ability to plan and explore based on spatial understanding and to execute those plans in accordance with the spatial layout of the environment.
This category evaluates an agent’s ability to understand the structural layout of objects and the positional relationships between landmarks in the environment.
Examples in each task type in 360CityArena. Environment Understanding includes Localization and Landmark Search with Language or Image; Path Reasoning includes Map Navigation and Vision-Language Navigation; Spatial Reasoning includes Object Count and Relational Spatial Reasoning.
We compare six closed and eight open models across all 175 tasks. GPT-6 Astra achieves the highest model mean score, while participants familiar with Akihabara outperform every model on all seven task types.
Local experts
77.3%
GPT-6 Astra
43.8%
The strongest evaluated model trails the human baseline by 33.5 percentage points. The human baseline comprises five participants familiar with Akihabara.
| Agent | Environment Understanding | Path Reasoning | Spatial Reasoning | Mean | ||||
|---|---|---|---|---|---|---|---|---|
| Loc. | Landmark (Lang.) |
Landmark (Img.) |
Map Nav. | VLN | Obj. Count | Rel. Reason. |
||
| Closed models | ||||||||
| GPT-5 | 8.0 | 16.0 | 48.0 | 0.0 | 8.0 | 2.4 | 36.0 | 16.9 |
| Claude Sonnet 4.5 | 4.0 | 4.0 | 16.0 | 4.0 | 4.0 | 10.8 | 4.0 | 6.7 |
| Gemini 2.5 Flash | 12.0 | 28.0 | 36.0 | 0.0 | 8.0 | 24.0 | 8.0 | 16.6 |
| GPT-6.1 Sol | 24.0 | 24.0 | 40.0 | 24.0 | 44.0 | 6.8 | 40.0 | 29.0 |
| GPT-6 Astra | 40.0 | 52.0 | 56.0 | 44.0 | 52.0 | 6.8 | 56.0 | 43.8 |
| Claude Sonnet 5.5 | 32.0 | 20.0 | 44.0 | 16.0 | 40.0 | 27.6 | 24.0 | 29.1 |
| Open models | ||||||||
| Qwen2.5-VL-32B-Instruct | 4.0 | 16.0 | 20.0 | 0.0 | 0.0 | 18.8 | 4.0 | 9.0 |
| InternVL3.5-8B | 8.0 | 16.0 | 12.0 | 0.0 | 4.0 | 2.0 | 0.0 | 6.0 |
| InternVL3.5-38B | 4.0 | 4.0 | 12.0 | 0.0 | 8.0 | 20.4 | 0.0 | 6.9 |
| Qwen3.5-9B | 4.0 | 24.0 | 0.0 | 0.0 | 8.0 | 8.0 | 4.0 | 6.9 |
| Qwen3.5-27B | 8.0 | 20.0 | 4.0 | 0.0 | 24.0 | 9.6 | 4.0 | 9.9 |
| Qwen3.8-27B | 0.0 | 12.0 | 20.0 | 0.0 | 20.0 | 10.4 | 16.0 | 11.2 |
| Gemma-4-12B-it | 0.0 | 16.0 | 32.0 | 0.0 | 8.0 | 7.6 | 0.0 | 9.1 |
| Gemma-4-31B-it | 8.0 | 20.0 | 44.0 | 0.0 | 28.0 | 30.8 | 16.0 | 21.0 |
| Human baseline | ||||||||
| Local experts | 68.0 | 64.0 | 92.0 | 92.0 | 88.0 | 45.2 | 92.0 | 77.3 |
Overall results (%). Each task type contains 25 tasks. Scores are success rate (SR) for six task types and mean relative accuracy (MRA) for Object Count. Mean is the equal-weight average of the seven unrounded scores. Unanswered tasks receive zero. Bold values indicate the best model score in each column.
Model agents use a memory-augmented ReAct-style framework. Generation settings vary by model.
Object Count uses MRA over ten thresholds from 0.50 to 0.95 in increments of 0.05.
Relational Spatial Reasoning uses a shared semantic-match judge
(Azure OpenAI gpt-6-luna, reasoning effort medium).
Empty and unsubmitted answers receive zero and remain in the denominator.
The human baseline measures task performance by five participants familiar with Akihabara; it does not represent the general population. See the evaluation protocol for task scoring rules.
GPT-6 Astra leads with a mean score of 43.8%, compared with 77.3% for local experts. It achieves the best model score on six of seven task types. Gemma-4-31B-it leads the open models with a mean score of 21.0% and achieves the highest model Object Count score, 30.8%.
Image-based landmark search outperforms language-based search in 11 of 14 models. For example, GPT-6 Astra scores 56.0% vs. 52.0%, and Gemma-4-31B-it scores 44.0% vs. 20.0%. The pattern does not hold for InternVL3.5-8B or Qwen3.5-9B and 27B.
All eight open models score 0.0% on Map Navigation. The best model, GPT-6 Astra, reaches 44.0%, compared with 92.0% for local experts. Following a map through the captured city remains difficult even for models that succeed at finding landmarks.
@inproceedings{watanabe2026360cityarena,
title = {360CityArena: A Realistic Virtual Urban Navigation Benchmark for Embodied Agents},
author = {Watanabe, Kenta and Miyai, Atsuyuki and Takenawa, Mizuki and Aizawa, Kiyoharu and Yamasaki, Toshihiko},
booktitle = {Proceedings of the European Conference on Computer Vision (ECCV)},
year = {2026}
}@article{takenawa2026building,
title = {Building and Evaluating a Realistic Virtual World for Large Scale Urban Exploration from 360° Videos},
author = {Takenawa, Mizuki and Sugimoto, Naoki and W{\"o}hler, Leslie and Ikehata, Satoshi and Aizawa, Kiyoharu},
journal = {Multimedia Tools and Applications},
volume = {85},
number = {2},
pages = {149},
year = {2026}
}