DragNUWA: Fine-grained Control in Video Generation by Integrating Text, Image, and Trajectory

Understand

Controllable video generation has gained significant attention in recent years.

• However, two main limitations persist: Firstly, most existing works focus on either text, image, or trajectory-based control, leading to an inability to achieve fine-grained control in videos.
• Secondly, trajectory control research is still in its early stages, with most experiments being conducted on simple datasets like Human3.6M.
• This constraint limits the models' capability to process open-domain images and effectively handle complex curved trajectories.