2023

AltDiffusion: A Multilingual Text-to-Image Diffusion Model

Ye, Fulong, Liu, Guang, Wu, Xinya et al.

Understand

Large Text-to-Image(T2I) diffusion models have shown a remarkable capability to produce photorealistic and diverse images based on text inputs.

However, existing works only support limited language input, e.g., English, Chinese, and Japanese, leaving users beyond these languages underserved and blocking the global expansion of T2I models.
Therefore, this paper presents AltDiffusion, a novel multilingual T2I diffusion model that supports eighteen different languages.
Specifically, we first train a multilingual text encoder based on the knowledge distillation.

Built on

Unsupervised cross-lingual representation learning at scale
Original
Conneau, A.; Khandelwal, K.; Goyal, N.; Chaudhary, V.; Wenzek, G.; Guzmán, F.; Grave, E.; Ott, M.; Zettlemoyer, L.; and Stoyanov, V. 2019 · 1911
Earlier work this paper cites.
Multilingual translation with extensible multilingual pretraining and finetuning
Original
Tang, Y.; Tran, C.; Li, X.; Chen, P.-J.; Goyal, N.; Chaudhary, V.; Gu, J.; and Fan, A. 2020 · 2008
Earlier work this paper cites.
Pp-ocr: A practical ultra lightweight ocr system
Original
Du, Y.; Li, C.; Guo, R.; Yin, X.; Liu, W.; Zhou, J.; Bai, Y.; Yu, Z.; Yang, Y.; Dang, Q.; et al. 2020 · 2009
Earlier work this paper cites.
Towards zero-shot Cross-lingual Image retrieval
Original
Aggarwal, P.; and Kale, A. 2020 · 2012
Earlier work this paper cites.
Microsoft coco: Common objects in context
Lin, T.-Y.; Maire, M.; Belongie, S.; Hays, J.; Perona, P.; Ramanan, D.; Dollár, P.; and Zitnick, C. L. 2014 · 2014
Earlier work this paper cites.
U-net: Convolutional networks for biomedical image segmentation
Ronneberger, O.; Fischer, P.; and Brox, T. 2015 · 2015
Earlier work this paper cites.
Multi30K: Multilingual English-German Image Descriptions
Elliott, D.; Frank, S.; Sima’an, K.; and Specia, L. 2016 · 2016
Earlier work this paper cites.
Improved techniques for training gans
Salimans, T.; Goodfellow, I.; Zaremba, W.; Cheung, V.; Radford, A.; and Chen, X. 2016 · 2016
Earlier work this paper cites.
Gans trained by a two time-scale update rule converge to a local nash equilibrium
Heusel, M.; Ramsauer, H.; Unterthiner, T.; Nessler, B.; and Hochreiter, S. 2017 · 2017
Earlier work this paper cites.
Decoupled weight decay regularization
Original
Loshchilov, I.; and Hutter, F. 2017 · 2017
Earlier work this paper cites.
Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models
Plummer, B. A.; Wang, L.; Cervantes, C. M.; Caicedo, J. C.; Hockenmaier, J.; and Lazebnik, S. 2017 · 2017
Earlier work this paper cites.

Similar

A Dataset and Reranking Method for Multimodal MT of User-Generated Image Captions
Schamoni, S.; Hitschler, J.; and Riezler, S. 2018 · 2018
Cited alongside, same era.
Denoising diffusion probabilistic models
Ho, J.; Jain, A.; and Abbeel, P. 2020 · 2020
Cited alongside, same era.
CogView: Mastering Text-to-Image Generation via Transformers
Original
Ding, M.; Yang, Z.; Hong, W.; Zheng, W.; Zhou, C.; Yin, D.; Lin, J.; Zou, X.; Shao, Z.; Yang, H.; and Tang, J. 2021 · 2021
Cited alongside, same era.
Clipscore: A reference-free evaluation metric for image captioning
Original
Hessel, J.; Holtzman, A.; Forbes, M.; Bras, R. L.; and Choi, Y. 2021 · 2021
Cited alongside, same era.
Lora: Low-rank adaptation of large language models
Original
Hu, E. J.; Shen, Y.; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; Wang, L.; and Chen, W. 2021 · 2021
Cited alongside, same era.
Wit: Wikipedia-based image text dataset for multimodal multilingual machine learning
Srinivasan, K.; Raman, K.; Chen, J.; Bendersky, M.; and Najork, M. 2021 · 2021
Cited alongside, same era.
ERNIE-ViLG: Unified generative pre-training for bidirectional vision-language generation
Original
Zhang, H.; Yin, W.; Fang, Y.; Li, L.; Duan, B.; Wu, Z.; Sun, Y.; Tian, H.; Wu, H.; and Wang, H. 2021 · 2021
Cited alongside, same era.
Cross-lingual and multilingual clip
Carlsson, F.; Eisen, P.; Rekathati, F.; and Sahlgren, M. 2022 · 2022
Cited alongside, same era.
AltCLIP: Altering the Language Encoder in CLIP for Extended Language Capabilities
Original
Chen, Z.; Liu, G.; Zhang, B.-W.; Ye, F.; Yang, Q.; and Wu, L. 2022 · 2022
Cited alongside, same era.
Reproducible scaling laws for contrastive language-image learning
Original
Cherti, M.; Beaumont, R.; Wightman, R.; Wortsman, M.; Ilharco, G.; Gordon, C.; Schuhmann, C.; Schmidt, L.; and Jitsev, J. 2022 · 2022
Cited alongside, same era.
CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformers
Original
Ding, M.; Zheng, W.; Hong, W.; and Tang, J. 2022 · 2022
Cited alongside, same era.
Japanese Stable Diffusion
Shing, M.; and Sawada, K. 2022a
Cited in the paper.

Then

ERNIE-ViLG 2.0: Improving Text-to-Image Diffusion Model with Knowledge-Enhanced Mixture-of-Denoising-Experts
Original
Feng, Z.; Zhang, Z.; Yu, X.; Fang, Y.; Li, L.; Chen, X.; Lu, Y.; Liu, J.; Yin, W.; Feng, S.; et al. 2022 · 2022
Later among the works it cites.
Classifier-free diffusion guidance
Original
Ho, J.; and Salimans, T. 2022 · 2022
Later among the works it cites.
Hierarchical text-conditional image generation with clip latents
Original
Ramesh, A.; Dhariwal, P.; Nichol, A.; Chu, C.; and Chen, M. 2022 · 2022
Later among the works it cites.
High-resolution image synthesis with latent diffusion models
Rombach, R.; Blattmann, A.; Lorenz, D.; Esser, P.; and Ommer, B. 2022 · 2022
Later among the works it cites.
Photorealistic text-to-image diffusion models with deep language understanding
Saharia, C.; Chan, W.; Saxena, S.; Li, L.; Whang, J.; Denton, E. L.; Ghasemipour, K.; Gontijo Lopes, R.; Karagol Ayan, B.; Salimans, T.; et al. 2022 · 2022
Later among the works it cites.
Laion-5b: An open large-scale dataset for training next generation image-text models
Original
Schuhmann, C.; Beaumont, R.; Vencu, R.; Gordon, C.; Wightman, R.; Cherti, M.; Coombes, T.; Katta, A.; Mullis, C.; Wortsman, M.; et al. 2022 · 2022
Later among the works it cites.
Crossmodal-3600: A massively multilingual multimodal evaluation dataset
Original
Thapliyal, A. V.; Pont-Tuset, J.; Chen, X.; and Soricut, R. 2022 · 2022
Later among the works it cites.
ERNIE-ViLG 2.0: Improving text-to-image diffusion model with knowledge-enhanced mixture-of-denoising-experts
Feng, Z.; Zhang, Z.; Yu, X.; Fang, Y.; Li, L.; Chen, X.; Lu, Y.; Liu, J.; Yin, W.; Feng, S.; et al. 2023 · 2023
Closest in time.
Erasing Concepts from Diffusion Models
Original
Gandikota, R.; Materzynska, J.; Fiotto-Kaufman, J.; and Bau, D. 2023 · 2023
Closest in time.
Adding Conditional Control to Text-to-Image Diffusion Models
Original
Zhang, L.; and Agrawala, M. 2023 · 2023
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…