2023

MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editing

Zhang, Kai, Mo, Lingbo, Chen, Wenhu et al.

Understand

Text-guided image editing is widely needed in daily life, ranging from personal use to professional applications such as Photoshop.

However, existing methods are either zero-shot or trained on an automatically synthesized dataset, which contains a high volume of noise.
Thus, they still require lots of manual tuning to produce desirable outcomes in practice.
To address this issue, we introduce MagicBrush (https://osu-nlp-group.github.io/MagicBrush/), the first large-scale, manually annotated dataset for instruction-guided real image editing that covers diverse scenarios: single-turn, multi-turn, mask-provided, and mask-free editing.

Built on

Image-based modeling and photo editing
B. M. Oh, M. Chen, J. Dorsey, and F. Durand · 2001
Earlier work this paper cites.
Automated flower classification over a large number of classes
M.-E. Nilsback and A. Zisserman · 2008
Earlier work this paper cites.
Microsoft COCO: common objects in context
T. Lin, M. Maire, S. J. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick · 2014
Earlier work this paper cites.
Caltech-ucsd birds 200
P. Welinder, S. Branson, T. Mita, C. Wah, F. Schroff, S. Belongie, and P. Perona · 2014
Earlier work this paper cites.
Learning deep representations of fine-grained visual descriptions
S. E. Reed, Z. Akata, H. Lee, and B. Schiele · 2016
Earlier work this paper cites.
Semantic image synthesis via adversarial learning
H. Dong, S. Yu, C. Wu, and Y. Guo · 2017
Earlier work this paper cites.
Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks
H. Zhang, T. Xu, and H. Li · 2017
Earlier work this paper cites.
Attngan: Fine-grained text to image generation with attentional generative adversarial networks
T. Xu, P. Zhang, Q. Huang, H. Zhang, Z. Gan, X. Huang, and X. He · 2018
Earlier work this paper cites.
Tell, draw, and repeat: Generating and modifying images based on continual linguistic instruction
A. El-Nouby, S. Sharma, H. Schulz, R. D. Hjelm, L. E. Asri, S. E. Kahou, Y. Bengio, and G. W. Taylor · 2019
Earlier work this paper cites.
Language models are few-shot learners
T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei · 2020
Earlier work this paper cites.
SSCR: iterative language-based image editing via self-supervised counterfactual reasoning
T. Fu, X. Wang, S. T. Grafton, M. P. Eckstein, and W. Y. Wang · 2020
Earlier work this paper cites.
Denoising diffusion probabilistic models
J. Ho, A. Jain, and P. Abbeel · 2020
Earlier work this paper cites.
Manigan: Text-guided image manipulation
B. Li, X. Qi, T. Lukasiewicz, and P. H. S. Torr · 2020
Earlier work this paper cites.
Lightweight generative adversarial networks for text-guided image manipulation
B. Li, X. Qi, P. H. S. Torr, and T. Lukasiewicz · 2020
Earlier work this paper cites.

Similar

Open-edit: Open-domain image manipulation with open-vocabulary instructions
X. Liu, Z. Lin, J. Zhang, H. Zhao, Q. Tran, X. Wang, and H. Li · 2020
Cited alongside, same era.
Paint by word
Original
D. Bau, A. Andonian, A. Cui, Y. Park, A. Jahanian, A. Oliva, and A. Torralba · 2021
Cited alongside, same era.
Emerging properties in self-supervised vision transformers
M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin · 2021
Cited alongside, same era.
An image is worth 16x16 words: Transformers for image recognition at scale
A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby · 2021
Cited alongside, same era.
Taming transformers for high-resolution image synthesis
P. Esser, R. Rombach, and B. Ommer · 2021
Cited alongside, same era.
Editgan: High-precision semantic image editing
H. Ling, K. Kreis, D. Li, S. W. Kim, A. Torralba, and S. Fidler · 2021
Cited alongside, same era.
Learning transferable visual models from natural language supervision
A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever · 2021
Cited alongside, same era.
Denoising diffusion implicit models
J. Song, C. Meng, and S. Ermon · 2021
Cited alongside, same era.
Text as neural operator: Image manipulation by text instruction
T. Zhang, H. Tseng, L. Jiang, W. Yang, H. Lee, and I. Essa · 2021
Cited alongside, same era.
Blended diffusion for text-driven editing of natural images
O. Avrahami, D. Lischinski, and O. Fried · 2022
Cited alongside, same era.
Text2live: Text-driven layered image and video editing
O. Bar-Tal, D. Ofri-Amar, R. Fridman, Y. Kasten, and T. Dekel · 2022
Cited alongside, same era.
Instructpix2pix: Learning to follow image editing instructions
Original
T. Brooks, A. Holynski, and A. A. Efros · 2022
Cited alongside, same era.

Then

Imagic: Text-based real image editing with diffusion models
Original
B. Kawar, S. Zada, O. Lang, O. Tov, H. Chang, T. Dekel, I. Mosseri, and M. Irani · 2022
Later among the works it cites.
SDEdit: Guided image synthesis and editing with stochastic differential equations
C. Meng, Y. He, Y. Song, J. Song, J. Wu, J.-Y. Zhu, and S. Ermon · 2022
Later among the works it cites.
Null-text inversion for editing real images using guided diffusion models
Original
R. Mokady, A. Hertz, K. Aberman, Y. Pritch, and D. Cohen-Or · 2022
Later among the works it cites.
GLIDE: towards photorealistic image generation and editing with text-guided diffusion models
A. Q. Nichol, P. Dhariwal, A. Ramesh, P. Shyam, P. Mishkin, B. McGrew, I. Sutskever, and M. Chen · 2022
Later among the works it cites.
Chatgpt, 2022
OpenAI · 2022
Later among the works it cites.
Training language models to follow instructions with human feedback
Original
L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. F. Christiano, J. Leike, and R. Lowe · 2022
Later among the works it cites.
Hierarchical text-conditional image generation with CLIP latents
Original
A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen · 2022
Later among the works it cites.
High-resolution image synthesis with latent diffusion models
R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer · 2022
Later among the works it cites.
Semanticstylegan: Learning compositional generative priors for controllable image synthesis and editing
Y. Shi, X. Yang, Y. Wan, and X. Shen · 2022
Later among the works it cites.
Imagen editor and editbench: Advancing and evaluating text-guided image inpainting
Original
S. Wang, C. Saharia, C. Montgomery, J. Pont-Tuset, S. Noy, S. Pellegrini, Y. Onoe, S. Laszlo, D. J. Fleet, R. Soricut, J. Baldridge, M. Norouzi, P. Anderson, and W. Chan · 2022
Later among the works it cites.
Subject-driven text-to-image generation via apprenticeship learning, 2023
W. Chen, H. Hu, Y. Li, N. Ruiz, X. Jia, M.-W. Chang, and W. W. Cohen · 2023
Closest in time.
Is prompt all you need? no. a comprehensive and broader view of instruction learning
Original
R. Lou, K. Zhang, and W. Yin · 2023
Closest in time.
Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation
N. Ruiz, Y. Li, V. Jampani, Y. Pritch, M. Rubinstein, and K. Aberman · 2023
Closest in time.
HIVE: harnessing human feedback for instructional visual editing
Original
S. Zhang, X. Yang, Y. Feng, C. Qin, C. Chen, N. Yu, Z. Chen, H. Wang, S. Savarese, S. Ermon, C. Xiong, and R. Xu · 2023
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…