2023

Shape-aware Text-driven Layered Video Editing

Lee, Yao-Chih, Jang, Ji-Ze Genevieve, Chen, Yi-Ting et al.

Understand

Temporal consistency is essential for video editing applications.

Existing work on layered representation of videos allows propagating edits consistently to each frame.
These methods, however, can only edit object appearance rather than object shape changes due to the limitation of using a fixed UV mapping field for texture atlas.
We present a shape-aware, text-driven video editing method to tackle this challenge.

Built on

Principal warps: Thin-plate splines and the decomposition of deformations
Fred L. Bookstein · 1989
Earlier work this paper cites.
Flow-based video synthesis and editing
Kiran S Bhat, Steven M Seitz, Jessica K Hodgins, and Pradeep K Khosla · 2004
Earlier work this paper cites.
Spatial transformer networks
Max Jaderberg, Karen Simonyan, Andrew Zisserman, et al · 2015
Earlier work this paper cites.
Temporally coherent completion of dynamic video
Jia-Bin Huang, Sing Bing Kang, Narendra Ahuja, and Johannes Kopf · 2016
Earlier work this paper cites.
Generative adversarial text to image synthesis
Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee · 2016
Earlier work this paper cites.
Layerbuilder: Layer decomposition for interactive image and video color editing
Original
Sharon Lin, Matthew Fisher, Angela Dai, and Pat Hanrahan · 2017
Earlier work this paper cites.
The 2017 davis challenge on video object segmentation
Original
Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alex Sorkine-Hornung, and Luc Van Gool · 2017
Earlier work this paper cites.
Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks
Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris N Metaxas · 2017
Earlier work this paper cites.
Learning blind video temporal consistency
Wei-Sheng Lai, Jia-Bin Huang, Oliver Wang, Eli Shechtman, Ersin Yumer, and Ming-Hsuan Yang · 2018
Earlier work this paper cites.
Text-adaptive generative adversarial networks: manipulating images with natural language
Seonghyeon Nam, Yunji Kim, and Seon Joo Kim · 2018
Earlier work this paper cites.
Attngan: Fine-grained text to image generation with attentional generative adversarial networks
Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He · 2018
Earlier work this paper cites.
Stylizing video by example
Ondřej Jamriška, Šárka Sochorová, Ondřej Texler, Michal Lukáč, Jakub Fišer, Jingwan Lu, Eli Shechtman, and Daniel Sỳkora · 2019
Earlier work this paper cites.
Object-driven text-to-image synthesis via adversarial training
Wenbo Li, Pengchuan Zhang, Lei Zhang, Qiuyuan Huang, Xiaodong He, Siwei Lyu, and Jianfeng Gao · 2019
Earlier work this paper cites.
Flow-edge guided video completion
Chen Gao, Ayush Saraf, Jia-Bin Huang, and Johannes Kopf · 2020
Earlier work this paper cites.
Denoising diffusion probabilistic models
Jonathan Ho, Ajay Jain, and Pieter Abbeel · 2020
Earlier work this paper cites.
Manigan: Text-guided image manipulation
Bowen Li, Xiaojuan Qi, Thomas Lukasiewicz, and Philip HS Torr · 2020
Earlier work this paper cites.
Layered neural rendering for retiming people in video
Erika Lu, Forrester Cole, Tali Dekel, Weidi Xie, Andrew Zisserman, David Salesin, William T Freeman, and Michael Rubinstein · 2020
Earlier work this paper cites.
Denoising diffusion implicit models
Original
Jiaming Song, Chenlin Meng, and Stefano Ermon · 2020
Earlier work this paper cites.
Diffusion models beat gans on image synthesis
Prafulla Dhariwal and Alexander Nichol · 2021
Earlier work this paper cites.
Clipdraw: Exploring text-to-drawing synthesis through language-image encoders
Original
Kevin Frans, Lisa B Soros, and Olaf Witkowski · 2021
Earlier work this paper cites.

Similar

Layered neural atlases for consistent video editing
Yoni Kasten, Dolev Ofri, Oliver Wang, and Tali Dekel · 2021
Cited alongside, same era.
Fusedream: Training-free text-to-image generation with improved clip+ gan space optimization
Original
Xingchao Liu, Chengyue Gong, Lemeng Wu, Shujian Zhang, Hao Su, and Qiang Liu · 2021
Cited alongside, same era.
Omnimatte: Associating objects and their effects in video
Erika Lu, Forrester Cole, Tali Dekel, Andrew Zisserman, William T Freeman, and Michael Rubinstein · 2021
Cited alongside, same era.
Styleclip: Text-driven manipulation of stylegan imagery
Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski · 2021
Cited alongside, same era.
Learning transferable visual models from natural language supervision
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al · 2021
Cited alongside, same era.
Zero-shot text-to-image generation
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever · 2021
Cited alongside, same era.
Zero-shot text-to-image generation
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever · 2021
Cited alongside, same era.
Tedigan: Text-guided diverse face image generation and manipulation
Weihao Xia, Yujiu Yang, Jing-Hao Xue, and Baoyuan Wu · 2021
Cited alongside, same era.
ediffi: Text-to-image diffusion models with an ensemble of expert denoisers
Original
Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, et al · 2022
Cited alongside, same era.
Text2live: Text-driven layered image and video editing
Omer Bar-Tal, Dolev Ofri-Amar, Rafail Fridman, Yoni Kasten, and Tali Dekel · 2022
Cited alongside, same era.
Generating long videos of dynamic scenes
Tim Brooks, Janne Hellsten, Miika Aittala, Ting-Chun Wang, Timo Aila, Jaakko Lehtinen, Ming-Yu Liu, Alexei A Efros, and Tero Karras · 2022
Cited alongside, same era.
Vqgan-clip: Open domain image generation and editing with natural language guidance
Katherine Crowson, Stella Biderman, Daniel Kornis, Dashiell Stander, Eric Hallahan, Louis Castricato, and Edward Raff · 2022
Cited alongside, same era.

Then

Towards an end-to-end framework for flow-guided video inpainting
Zhen Li, Cheng-Ze Lu, Jianhua Qin, Chun-Le Guo, and Ming-Ming Cheng · 2022
Later among the works it cites.
Text to image generation with semantic-spatial aware gan
Wentong Liao, Kai Hu, Michael Ying Yang, and Bodo Rosenhahn · 2022
Later among the works it cites.
Deepfacevideoediting: Sketch-based deep editing of face videos
Feng-Lin Liu, Shu-Yu Chen, Yukun Lai, Chunpeng Li, Yue-Ren Jiang, Hongbo Fu, and Lin Gao · 2022
Later among the works it cites.
Text-driven stylization of video objects
Original
Sebastian Loeschcke, Serge Belongie, and Sagie Benaim · 2022
Later among the works it cites.
Glide: Towards photorealistic image generation and editing with text-guided diffusion models
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen · 2022
Later among the works it cites.
Dreamfusion: Text-to-3d using 2d diffusion
Original
Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall · 2022
Later among the works it cites.
Film: Frame interpolation for large motion
Fitsum Reda, Janne Kontkanen, Eric Tabellion, Deqing Sun, Caroline Pantofaru, and Brian Curless · 2022
Later among the works it cites.
High-resolution image synthesis with latent diffusion models
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer · 2022
Later among the works it cites.
Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation
Original
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman · 2022
Later among the works it cites.
Photorealistic text-to-image diffusion models with deep language understanding
Original
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al · 2022
Later among the works it cites.
Laion-5b: An open large-scale dataset for training next generation image-text models
Original
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al · 2022
Later among the works it cites.
Make-a-video: Text-to-video generation without text-video data
Original
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al · 2022
Later among the works it cites.
Stylegan-v: A continuous video generator with the price, image quality and perks of stylegan2
Ivan Skorokhodov, Sergey Tulyakov, and Mohamed Elhoseiny · 2022
Later among the works it cites.
Probabilistic warp consistency for weakly-supervised semantic correspondences
Prune Truong, Martin Danelljan, Fisher Yu, and Luc Van Gool · 2022
Later among the works it cites.
Phenaki: Variable length video generation from open domain textual description
Original
Ruben Villegas, Mohammad Babaeizadeh, Pieter-Jan Kindermans, Hernan Moraldo, Han Zhang, Mohammad Taghi Saffar, Santiago Castro, Julius Kunze, and Dumitru Erhan · 2022
Later among the works it cites.
Gan inversion for consistent video interpolation and manipulation
Original
Weihao Xia, Yujiu Yang, and Jing-Hao Xue · 2022
Later among the works it cites.
Temporally consistent semantic video editing
Original
Yiran Xu, Badour AlBahar, and Jia-Bin Huang · 2022
Later among the works it cites.
Predict, prevent, and evaluate: Disentangled text-driven image manipulation empowered by pre-trained vision-language model
Zipeng Xu, Tianwei Lin, Hao Tang, Fu Li, Dongliang He, Nicu Sebe, Radu Timofte, Luc Van Gool, and Errui Ding · 2022
Later among the works it cites.
Deformable sprites for unsupervised video decomposition
Vickie Ye, Zhengqi Li, Richard Tucker, Angjoo Kanazawa, and Noah Snavely · 2022
Later among the works it cites.
Generating videos with dynamics-aware implicit generative adversarial networks
Sihyun Yu, Jihoon Tack, Sangwoo Mo, Hyunsu Kim, Junho Kim, Jung-Woo Ha, and Jinwoo Shin · 2022
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…