Interview with William Yijiang Li: vision language models and the physical world

Summary

At the International Conference on Machine Learning (ICML 2026) , Dezhi Luo, Yijiang Li, Maijunxian Wang, Tianwei Zhao, Bingyang Wang, Siheng Wang, Pinyuan Feng, Pooyan Rahmanzadehgervi, Ziqiao Ma , and Hokin Deng presented their work Vision Language Models Cannot Reason About Physical Transformation . In this interview, William Yijiang Li tells us more about the research, the method the team used, and the controlled experiments they carried out. What is the topic of the research in your paper and why is it an interesting area for study? Our research asks a fairly fundamental question: do vision-language models actually understand how the physical world changes over time? Modern vision-language models are increasingly being used in areas such as robotics, embodied agents, and video understanding. They can recognize objects, describe scenes, and answer surprisingly sophisticated questions. However, being able to describe a scene is different from maintaining an understanding of an object or physical quantity as the scene changes. We study this through the concept of conservation . For example, if water is poured from a short, wide glass into a tall, narrow one, its appearance changes dramatically, but the amount of water remains the same. Similarly, spreading a row of coins farther apart does not change the number of coins. These tasks are simple for humans, but they require an important form of reasoning: you need to track what happens during a transformation and distinguish changes in appearance from changes in the underlying physical state. This is particularly interesting for embodied AI. If an AI system is expected to interact with the real world, it cannot rely only on static visual recognition. It needs to maintain and update representations of objects and their properties as events unfold. What are the main contributions of your research? I would summarize the contributions in three parts. First, and most importantly, our study shows that current vision-language models still struggle to maintain and update a stable representation of the physical world as it changes over time. Although these models can often answer basic questions about physical concepts such as conservation correctly, our controlled experiments show that this does not necessarily mean they understand the underlying transformation. Models that perform well when a quantity is conserved…