V-RAE: Rethinking Video Latent Spaces for Generation

Summary

V-RAE is a video representation autoencoder that builds compact generative latents on top of frozen vision foundation model representations. A lightweight temporal pooling module removes temporal redundancy while preserving semantic structure, and a video decoder reconstructs continuous motion from compressed features. On K600, V-RAE achieves an rFVD of 2.13, outperforming all evaluated large-scale pretrained video VAEs. Under matched generation settings, its best variant attains gFVD scores of 117.86 and 19.16 on UCF101 and K600, respectively, while converging up to 6x faster. The authors introduce tFVD, a temporal-coherence diagnostic that correlates more reliably with downstream generation quality. Additionally, V-RAE improves future video prediction on Cityscapes over the Wan 2.2 VAE latent space under matched settings. Experiments demonstrate that frozen semantic representations can support video reconstruction, generation, and predictive modeling.