2022

Imagen Editor and EditBench: Advancing and Evaluating Text-Guided Image Inpainting

Wang, Su, Saharia, Chitwan, Montgomery, Ceslee et al.

Understand

Text-guided image editing can have a transformative impact in supporting creative applications.

A key challenge is to generate edits that are faithful to input text prompts, while consistent with input images.
We present Imagen Editor, a cascaded diffusion model built, by fine-tuning Imagen on text-guided image inpainting.
Imagen Editor's edits are faithful to the text prompts, which is accomplished by using object detectors to propose inpainting masks during training.

Built on

Quick, simple and reliable: forced binary survey questions
Sara Dolnicar, Bettina Grün, and Friedrich Leisch · 2011
Earlier work this paper cites.
Ava: A large-scale database for aesthetic visual analysis
Naila Murray, Luca Marchesotti, and Florent Perronnin · 2012
Earlier work this paper cites.
Achieving accurate conclusions in evaluation of automatic machine translation metrics
Yvette Graham and Qun Liu · 2016
Earlier work this paper cites.
Sequence-Level Knowledge Distillation
Yoon Kim and Alexander M. Rush · 2016
Earlier work this paper cites.
Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, Michael Bernstein, and Li Fei-Fei · 2017
Earlier work this paper cites.
Inverted residuals and linear bottlenecks: Mobile networks for classification, detection and segmentation
Original
Mark Sandler, Andrew G. Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen · 2018
Earlier work this paper cites.
Generative image inpainting with contextual attention
Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang · 2018
Earlier work this paper cites.
Recognizing unseen attribute-object pair with generative model
Zhixiong Nan, Yang Liu, Nanning Zheng, and Song-Chun Zhu · 2019
Earlier work this paper cites.
Assessing Social and Intersectional Biases in Contextualized Word Representations
Yi Chern Tan and L. Elisa Celis · 2019
Earlier work this paper cites.
Free-form image inpainting with gated convolution
Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang · 2019
Earlier work this paper cites.
On the binding problem in artificial neural networks
Original
Klaus Greff, Sjoerd van Steenkiste, and Jürgen Schmidhuber · 2020
Earlier work this paper cites.
The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, Tom Duerig, and Vittorio Ferrari · 2020
Earlier work this paper cites.
Manigan: Text-guided image manipulation, 2020
Bowen Li, Xiaojuan Qi, Thomas Lukasiewicz, and Philip H. S. Torr · 2020
Earlier work this paper cites.
Bertscore: Evaluating text generation with bert
Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi · 2020
Earlier work this paper cites.
Paint by word
Original
David Bau, Alex Andonian, Audrey Cui, YeonHwan Park, Ali Jahanian, Aude Oliva, and Antonio Torralba · 2021
Earlier work this paper cites.
Multimodal datasets: misogyny, pornography, and malignant stereotypes
Original
Abeba Birhane, Vinay Uday Prabhu, and Emmanuel Kahembwe · 2021
Earlier work this paper cites.
Diffusion schrödinger bridge with applications to score-based generative modeling
Valentin De Bortoli, James Thornton, Jeremy Heng, and Arnaud Doucet · 2021
Earlier work this paper cites.
Clipscore: A reference-free evaluation metric for image captioning
Original
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi · 2021
Earlier work this paper cites.

Similar

Classifier-free diffusion guidance
Jonathan Ho and Tim Salimans · 2021
Cited alongside, same era.
Diffusionclip: Text-guided image manipulation using diffusion models
Original
Gwanghyun Kim and Jong Chul Ye · 2021
Cited alongside, same era.
Deduplicating training data makes language models better
Original
Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini · 2021
Cited alongside, same era.
GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models
Original
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Bob McGrew Pamela Mishkin, Ilya Sutskever, and Mark Chen · 2021
Cited alongside, same era.
Benchmark for compositional text-to-image synthesis
Dong Huk Park, Samaneh Azadi, Xihui Liu, Trevor Darrell, and Anna Rohrbach · 2021
Cited alongside, same era.
Learning transferable visual models from natural language supervision
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al · 2021
Cited alongside, same era.
Learning transferable visual models from natural language supervision
Original
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever · 2021
Cited alongside, same era.
Grave artifacts in image inpainting: Investigating the causes and untangling the factors, 2021
Walber Rodrigues, Felipe Walmsley, George Cavalcanti, Jonysberg Quintino, and Helder Pinho · 2021
Cited alongside, same era.
Palette: Image-to-Image Diffusion Models
Original
Chitwan Saharia, William Chan, Huiwen Chang, Chris A. Lee, Jonathan Ho, Tim Salimans, David J. Fleet, and Mohammad Norouzi · 2021
Cited alongside, same era.
Findings of the WMT 2021 shared task on quality estimation
Lucia Specia, Frédéric Blain, Marina Fomicheva, Chrysoula Zerva, Zhenhao Li, Vishrav Chaudhary, and André F. T. Martins · 2021
Cited alongside, same era.
Resolution-robust large mask inpainting with fourier convolutions
Original
Roman Suvorov, Elizaveta Logacheva, Anton Mashikhin, Anastasia Remizova, Arsenii Ashukha, Aleksei Silvestrov, Naejin Kong, Harshith Goka, Kiwoong Park, and Victor Lempitsky · 2021
Cited alongside, same era.
Rank-smoothed pairwise learning in perceptual quality assessment
Hossein Talebi, Ehsan Amid, Peyman Milanfar, and Manfred K. Warmuth · 2021
Cited alongside, same era.

Then

Make-a-scene: Scene-based text-to-image generation with human priors
Original
Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman · 2022
Closest in time.
Text-guided image inpainting
Ying Gao and Qing Zhu · 2022
Closest in time.
Prompt-to-Prompt Image Editing with Cross Attention Control
Original
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or · 2022
Closest in time.
Imagen video: High definition video generation with diffusion models
Original
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al · 2022
Closest in time.
Imagic: Text-based real image editing with diffusion models, 2022
Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani · 2022
Closest in time.
Leca: A learned approach for efficient cover-agnostic watermarking, 2022
Xiyang Luo, Michael Goebel, Elnaz Barshan, and Feng Yang · 2022
Closest in time.
Hierarchical Text-Conditional Image Generation with CLIP Latents
Original
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen · 2022
Closest in time.
High-Resolution Image Synthesis with Latent Diffusion Models
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer · 2022
Closest in time.
Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman · 2022
Closest in time.
Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi · 2022
Closest in time.
Photorealistic text-to-image diffusion models with deep language understanding, 2022
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi · 2022
Closest in time.
Image Super-Resolution via Iterative Refinement
Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi · 2022
Closest in time.
De-net: Dynamic text-guided image editing adversarial networks, 2022
Ming Tao, Bing-Kun Bao, Hao Tang, Fei Wu, Longhui Wei, and Qi Tian · 2022
Closest in time.
Unitune: Text-driven image editing by fine tuning an image generation model on a single image, 2022
Dani Valevski, Matan Kalman, Yossi Matias, and Yaniv Leviathan · 2022
Closest in time.
Manitrans: Entity-level text-guided image manipulation via token-wise semantic alignment and generation, 2022
Jianan Wang, Guansong Lu, Hang Xu, Zhenguo Li, Chunjing Xu, and Yanwei Fu · 2022
Closest in time.
Scaling Autoregressive Models for Content-Rich Text-to-Image Generation
Original
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, Ben Hutchinson, Wei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, and Yonghui Wu · 2022
Closest in time.
Interactive image generation with natural-language feedback
Yufan Zhou, Ruiyi Zhang, Jiuxiang Gu, Chris Tensmeyer, Tong Yu, Changyou Chen, Jinhui Xu, and Tong Sun · 2022
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…