AVA-Encoder: Towards Agent-Native Video Representation Learning

Summary

Creative agents still lack an effective way to learn from high-quality human films, limiting their ability to produce cinematic-grade videos. To address this, researchers propose the Agentic Video Auto-Encoder (AVA-Encoder), a framework for learning agent-native video representations via agentic auto-encoding. AVA-Encoder transforms a video into a knowledge graph (KG) representation and reconstructs it back into video. Its hierarchy and state nodes store structured text, while a linked asset layer holds generated images, audio, and video. Typed edges preserve relations between text descriptions and assets, enabling agents to understand, query, and edit. Reconstruction differences drive a textual-gradient optimization framework, with data-independent encoding policy pseudo-training in the outer loop and optional data-dependent KG refinement at test time. Experiments show AVA-Encoder improves by 20.7 percentage points over the strongest external baseline. In a controlled policy-only setting, its pseudo-trained shot-level Agentic Video Encoder policy outperforms a carefully human-tuned policy while using 74.3% fewer system-prompt tokens. The complete framework, a reliable video reconstruction benchmark, and the first dataset of high-quality film KG representations are released.