VideoPoet: A Large Language Model for Zero-Shot Video Generation

Understand

We present VideoPoet, a language model capable of synthesizing high-quality video, with matching audio, from a large variety of conditioning signals.

• VideoPoet employs a decoder-only transformer architecture that processes multimodal inputs -- including images, videos, text, and audio.
• The training protocol follows that of Large Language Models (LLMs), consisting of two stages: pretraining and task-specific adaptation.
• During pretraining, VideoPoet incorporates a mixture of multimodal generative objectives within an autoregressive Transformer framework.