Interactive World Simulator for Robot Policy Training and Evaluation

Summary

Traditional robot learning relies on collecting demonstrations and evaluating policies on real robots, which is slow, expensive, and hard to reproduce. We propose an Interactive World Simulator: a learned, action-conditioned video prediction model that predicts subsequent pixel frames given the current image and robot actions, without a physics engine. On a single RTX 4090, it runs for over 10 minutes at 15 FPS, producing stable and physically plausible video. This enables cheap data generation and scalable, reproducible policy evaluation. We trained the simulator on four tasks spanning different physical regimes: T pushing (rigid-body contact), rope routing (deformable-rigid interaction), mug grasping (fine-grained gripper dynamics), and pile sweeping (manipulating piles). All behaviors are learned solely from interaction data, with no hard-coded physics priors.