SemComp-Bench: Benchmarking Semantic Task Completion in Video Generation

Summary

We introduce Semantic Task Completion Video Generation, an outcome-oriented task requiring both the achievement of the intended outcome and semantic grounding, which characterizes the correspondence between the reference image and the generated outcome in terms of high-level task-relevant semantics. Evaluation focuses on the generated outcome and does not require a complete sequence of intermediate steps or appearance consistency. We construct SemComp-Data, an evaluation dataset covering six domains, with each instance consisting of a reference image, a detailed instruction, a brief instruction, and an outcome-centric video clip. A scalable four-stage curation pipeline converts raw videos into standardized instances. We further propose SemComp-Bench, an evaluation protocol using a vision-language model (VLM) to answer structured binary questions, reporting OA Score and GR Score for Outcome Achievement and Generation Reliability. Experiments on representative models show that achieving intended outcomes while maintaining task-relevant semantic grounding remains challenging.