ESI-Bench Catches Frontier AI Failing 3D Space: Seeing and Acting Diverge

Summary

A Stanford-led research team that includes Fei-Fei Li published a benchmark on May 18, 2026, that documents a specific, measurable failure in every frontier multimodal AI model tested: when forced to actively move through a 3D environment rather than interpret a pre-composed image, state-of-the-art systems consistently choose the wrong actions, skip viewpoints that would correct their mistakes, and commit to wrong answers with high confidence. The paper names this failure "action blindness." For robotics buyers, simulation engineers, and any developer whose product requires an AI agent to reason about physical space, the finding supplies a test that existing leaderboards do not. ESI-Bench Redefines What Spatial Intelligence Requires The benchmark, titled "ESI-Bench: Towards Embodied Spatial Intelligence that Closes the Perception-Action Loop," covers 10 task categories and 29 subcategories built inside the OmniGibson simulator on top of Stanford's BEHAVIOR-1K activity dataset. Its approximately 3,000 tasks are organized around what developmental psychologist Elizabeth Spelke at Harvard has identified as core knowledge systems — the domain-specific cognitive systems human infants use to reason about objects, agents, numbers, space, and form. The conceptual shift the benchmark makes is precise and consequential. Prior spatial-intelligence benchmarks assumed oracle observations: the model was handed the right view of the scene before being asked a question. ESI-Bench removes that assumption. The agent must first decide which ability to deploy — perception, locomotion, or manipulation — and in what sequence, in order to accumulate the evidence needed to answer. As described on the project page, the agent must actively uncover what is unseen: occluded structure, dynamics, containment, and functionality that no single static image can reveal. The task set reflects that design. Agents are asked to compare the liquid-holding capacity of two containers by manipulating them, predict whether a deformable object will conform to a surface, judge whether a tower will balance under given mass and geometry distributions, and distinguish a mirror reflection from the real scene by repositioning. Each task has a correct answer that is structurally inaccessible without choosing an action first. Frontier Models See But Do Not Know When to Move Experiments on state-of-the-art mu…