2024

Evaluating Text-to-Visual Generation with Image-to-Text Generation

Lin, Zhiqiu, Pathak, Deepak, Li, Baiqi et al.

Understand

Despite significant progress in generative AI, comprehensive evaluation remains challenging because of the lack of effective metrics and standardized benchmarks.

For instance, the widely-used CLIPScore measures the alignment between a (generated) image and text prompt, but it fails to produce reliable scores for complex prompts involving compositions of objects, attributes, and relations.
One reason is that text encoders of CLIP can notoriously act as a "bag of words", conflating prompts such as "the horse is eating the grass" with "the grass is eating the horse".
To address this, we introduce the VQAScore, which uses a visual-question-answering (VQA) model to produce an alignment score by computing the probability of a "Yes" answer to a simple "Does this figure show '{text}'?" question.

Built on

Multi-concept customization of text-to-image diffusion
Kumari, N., Zhang, B., Zhang, R., Shechtman, E., and Zhu, J.-Y. (2023) · 1941
Earlier work this paper cites.
View from the top: Hierarchies and reverse hierarchies in the visual system
Hochstein, S. and Ahissar, M. (2002) · 2002
Earlier work this paper cites.
A neural probabilistic language model
Bengio, Y., Ducharme, R., Vincent, P., and Jauvin, C. (2003) · 2003
Earlier work this paper cites.
Microsoft coco: Common objects in context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. L. (2014) · 2014
Earlier work this paper cites.
Improved techniques for training gans
Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., and Chen, X. (2016) · 2016
Earlier work this paper cites.
Making the v in vqa matter: Elevating the role of image understanding in visual question answering
Goyal, Y., Khot, T., Summers-Stay, D., Batra, D., and Parikh, D. (2017) · 2017
Earlier work this paper cites.
Gans trained by a two time-scale update rule converge to a local nash equilibrium
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. (2017) · 2017
Earlier work this paper cites.
A corpus of natural language for visual reasoning
Suhr, A., Lewis, M., Yeh, J., and Artzi, Y. (2017) · 2017
Earlier work this paper cites.
The unreasonable effectiveness of deep features as a perceptual metric
Zhang, R., Isola, P., Efros, A. A., Shechtman, E., and Wang, O. (2018) · 2018
Earlier work this paper cites.
Gqa: A new dataset for real-world visual reasoning and compositional question answering
Hudson, D. A. and Manning, C. D. (2019) · 2019
Earlier work this paper cites.
Exploring the limits of transfer learning with a unified text-to-text transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. (2020) · 2020
Earlier work this paper cites.
Clipscore: A reference-free evaluation metric for image captioning
Original
Hessel, J., Holtzman, A., Forbes, M., Bras, R. L., and Choi, Y. (2021) · 2021
Earlier work this paper cites.
Sdedit: Guided image synthesis and editing with stochastic differential equations
Original
Meng, C., He, Y., Song, Y., Song, J., Wu, J., Zhu, J.-Y., and Ermon, S. (2021) · 2021
Earlier work this paper cites.
Learning transferable visual models from natural language supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., · 2021
Earlier work this paper cites.
Pali: A jointly-scaled multilingual language-image model
Original
Chen, X., Wang, X., Changpinyo, S., Piergiovanni, A., Padlewski, P., Salz, D., Goodman, S., Grycner, A., Mustafa, B., Beyer, L., · 2022
Earlier work this paper cites.
Scaling instruction-finetuned language models
Original
Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., · 2022
Earlier work this paper cites.
Imagen video: High definition video generation with diffusion models
Original
Ho, J., Chan, W., Saharia, C., Whang, J., Gao, R., Gritsenko, A., Kingma, D. P., Poole, B., Norouzi, M., Fleet, D. J., · 2022
Earlier work this paper cites.
Compositional visual generation with composable diffusion models
Liu, N., Li, S., Du, Y., Torralba, A., and Tenenbaum, J. B. (2022) · 2022
Earlier work this paper cites.
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., · 2022
Earlier work this paper cites.
Dreamfusion: Text-to-3d using 2d diffusion
Original
Poole, B., Jain, A., Barron, J. T., and Mildenhall, B. (2022) · 2022
Earlier work this paper cites.
High-resolution image synthesis with latent diffusion models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. (2022) · 2022
Earlier work this paper cites.

Similar

Photorealistic text-to-image diffusion models with deep language understanding
Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E. L., Ghasemipour, K., Gontijo Lopes, R., Karagol Ayan, B., Salimans, T., · 2022
Cited alongside, same era.
Make-a-video: Text-to-video generation without text-video data
Original
Singer, U., Polyak, A., Hayes, T., Yin, X., An, J., Zhang, S., Hu, Q., Yang, H., Ashual, O., Gafni, O., · 2022
Cited alongside, same era.
Unifying language learning paradigms
Original
Tay, Y., Dehghani, M., Tran, V. Q., Garcia, X., Bahri, D., Schuster, T., Zheng, H. S., Houlsby, N., and Metzler, D. (2022) · 2022
Cited alongside, same era.
Winoground: Probing vision and language models for visio-linguistic compositionality
Thrush, T., Jiang, R., Bartolo, M., Singh, A., Williams, A., Kiela, D., and Ross, C. (2022) · 2022
Cited alongside, same era.
Scaling autoregressive models for content-rich text-to-image generation
Original
Yu, J., Xu, Y., Koh, J. Y., Luong, T., Baid, G., Wang, Z., Vasudevan, V., Ku, A., Yang, Y., Ayan, B. K., · 2022
Cited alongside, same era.
When and why vision-language models behave like bags-of-words, and what to do about it?
Yuksekgonul, M., Bianchi, F., Kalluri, P., Jurafsky, D., and Zou, J. (2022) · 2022
Cited alongside, same era.
Improving image generation with better captions
Betker, J., Goh, G., Jing, L., Brooks, T., Wang, J., Li, L., Ouyang, L., Zhuang, J., Lee, J., Guo, Y., · 2023
Cited alongside, same era.
Instructpix2pix: Learning to follow image editing instructions
Brooks, T., Holynski, A., and Efros, A. A. (2023) · 2023
Cited alongside, same era.
Muse: Text-to-image generation via masked generative transformers
Original
Chang, H., Zhang, H., Barber, J., Maschinot, A., Lezama, J., Jiang, L., Yang, M.-H., Murphy, K., Freeman, W. T., Rubinstein, M., · 2023
Cited alongside, same era.
Instructeval: Towards holistic evaluation of instruction-tuned large language models
Original
Chia, Y. K., Hong, P., Bing, L., and Poria, S. (2023) · 2023
Cited alongside, same era.
Ties matter: Meta-evaluating modern metrics with pairwise accuracy and tie calibration
Deutsch, D., Foster, G., and Freitag, M. (2023) · 2023
Cited alongside, same era.
Visual programming: Compositional visual reasoning without training
Gupta, T. and Kembhavi, A. (2023) · 2023
Cited alongside, same era.

Then

Llmscore: Unveiling the power of large language models in text-to-image synthesis evaluation
Original
Lu, Y., Yang, X., Li, X., Wang, X. E., and Wang, W. Y. (2023) · 2023
Later among the works it cites.
Crepe: Can vision-language foundation models reason compositionally?
Ma, Z., Hong, J., Gul, M. O., Gandhi, M., Gao, I., and Krishna, R. (2023) · 2023
Later among the works it cites.
Latent-nerf for shape-guided generation of 3d shapes and textures
Metzer, G., Richardson, E., Patashnik, O., Giryes, R., and Cohen-Or, D. (2023) · 2023
Later among the works it cites.
Gpt-4 technical report
Original
OpenAI (2023) · 2023
Later among the works it cites.
Toward verifiable and reproducible human evaluation for text-to-image generation
Otani, M., Togashi, R., Sawai, Y., Ishigami, R., Nakashima, Y., Rahtu, E., Heikkilä, J., and Satoh, S. (2023) · 2023
Later among the works it cites.
Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation
Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., and Aberman, K. (2023) · 2023
Later among the works it cites.
Mvdream: Multi-view diffusion for 3d generation
Original
Shi, Y., Wang, P., Ye, J., Long, M., Li, K., and Yang, X. (2023) · 2023
Later among the works it cites.
Divide, evaluate, and refine: Evaluating and improving text-to-image alignment with iterative vqa feedback
Original
Singh, J. and Zheng, L. (2023) · 2023
Later among the works it cites.
Dreamsync: Aligning text-to-image generation with image understanding feedback
Original
Sun, J., Fu, D., Hu, Y., Wang, S., Rassin, R., Juan, D.-C., Alon, D., Herrmann, C., van Steenkiste, S., Krishna, R., · 2023
Later among the works it cites.
Vipergpt: Visual inference via python execution for reasoning
Original
Surís, D., Menon, S., and Vondrick, C. (2023) · 2023
Later among the works it cites.
Llama 2: Open foundation and fine-tuned chat models
Original
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., · 2023
Later among the works it cites.
Imagereward: Learning and evaluating human preferences for text-to-image generation
Original
Xu, J., Liu, X., Wu, Y., Tong, Y., Li, Q., Ding, M., Tang, J., and Dong, Y. (2023) · 2023
Later among the works it cites.
Idea2img: Iterative self-refinement with gpt-4v (ision) for automatic image design and generation
Original
Yang, Z., Wang, J., Li, L., Lin, K., Lin, C.-C., Liu, Z., and Wang, L. (2023) · 2023
Later among the works it cites.
What you see is what you read? improving text-image alignment evaluation
Original
Yarom, M., Bitton, Y., Changpinyo, S., Aharoni, R., Herzig, J., Lang, O., Ofek, E., and Szpektor, I. (2023) · 2023
Later among the works it cites.
Deepfloyd IF
Deepfloyd IF (2024) · 2024
Closest in time.
GenAI-bench: A holistic benchmark for compositional text-to-visual generation
Li, B., Lin, Z., Pathak, D., Li, J. E., Xia, X., Neubig, G., Zhang, P., and Ramanan, D. (2024b) · 2024
Closest in time.
Revisiting the role of language priors in vision-language models
Original
Lin, Z., Chen, X., Pathak, D., Zhang, P., and Ramanan, D. (2024) · 2024
Closest in time.
Language models as black-box optimizers for vision-language models
Original
Liu, S., Lin, Z., Yu, S., Lee, R., Ling, T., Pathak, D., and Ramanan, D. (2024) · 2024
Closest in time.
Midjourney
Midjourney (2024) · 2024
Closest in time.
The neglected tails of vision-language models
Original
Parashar, S., Lin, Z., Liu, T., Dong, X., Li, Y., Ramanan, D., Caverlee, J., and Kong, S. (2024) · 2024
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…