Import AI 476: Humanity’s Sixth Sense benchmark shows AI visual reasoning lags humans

Summary

Welcome to Import AI, a newsletter about AI research. Import AI runs on arXiv, cappuccinos, and feedback from readers. If you’d like to support this, please subscribe. Subscribe now New benchmark shows how AI models lack intuitive visual reasoning: …Humanity’s Sixth Sense highlights some shortcomings of current AI systems… AI startups Elorian and Scale AI have built and released Humanity’s Sixth Sense, a benchmark that “measures how a model interprets and reasons through information a visual scene implies but doesn’t explicitly show.” Some examples of the types of task in HSS are a picture of a bookcase and being asked whether it can fit more books, or a photo of horses pulling a plough and a question of whether the chains bear the plough’s drag weight. HSS contains 522 questions that cover 288 images and 234 short video clips (totalling 17.6 hours of video). The questions fall into four categories: Temporal & causal dynamics: “What happened before the scene, and what happens next.” Physical & spatial logic: “What fits, what’s in reach, and what someone else can see from where they are standing.” Social understanding : “Intent, attention, belonging.” Abstract & contextual inference : “The patterns and rules a scene implies.” Results: The human baseline for this is not perfect, but is very high, with people scoring aggregate pass@1 of 93.1, broken down across four categories: Image (94.1), video (91.9), time (97), and space (94.9). By comparison, the best scoring AI system, GPT 6 Astra, gets pass@1 of 53.6, across image (55.6), video (51.1), time (58.2), and space (53.6). Interestingly, Chinese open weight models do poorly here, with GLM-5.3-flash getting a pass@1 of 26.8, and Kimi K3 getting pass@1 of 25.5. “The two most common single mistakes were (1) missing a detail that decides the answer (21% of failures) and (2) misidentifying an object, person, or role (20%),” they write. Why this matters - HSS tells us whether world models are emergent: One way of looking at HSS is highlighting deficiencies in current systems, but a more interesting way is highlighting how adept today’s frontier LLMs are at doing tasks which many people have long assumed would require sophisticated, different AI systems to solve. For instance, many people have claimed that true visual reasoning will rely on developing spatially aware 2D and 3D world models, then using that for reaso…