2019

Semantic Object Accuracy for Generative Text-to-Image Synthesis

Hinz, Tobias, Heinrich, Stefan, Wermter, Stefan

Understand

Generative adversarial networks conditioned on textual image descriptions are capable of generating realistic-looking images.

However, current methods still struggle to generate images based on complex image captions from a heterogeneous domain.
Furthermore, quantitatively evaluating these text-to-image models is challenging, as most evaluation metrics only judge image quality but not the conformity between the image and its caption.
To address these challenges we introduce a new model that explicitly models individual objects within an image and a new evaluation metric called Semantic Object Accuracy (SOA) that specifically evaluates images given an image caption.

Built on

J. Giménez and L. Màrquez, “Linguistic features for automatic evaluation of heterogenous mt systems,” in Proc. of the ACL Workshop on Statistical Machine Translation . Association for Computational Linguistics, 2007, pp. 256–264
2007
Earlier work this paper cites.
I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in Proc. Advances Neural Inf. Process. Syst. , 2014, pp. 2672–2680
2014
Earlier work this paper cites.
T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in Proc. Europ. Conf. Comput. Vis. , 2014, pp. 740–755
2014
Earlier work this paper cites.
M. Mirza and S. Osindero, “Conditional generative adversarial nets,” arXiv preprint arXiv:1411.1784 , 2014
Original
2014
Earlier work this paper cites.
B. Efron, “Bootstrap methods: another look at the jackknife,” in Breakthroughs in statistics . Springer, 1992, pp. 569–593
2014
Earlier work this paper cites.
T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen, “Improved techniques for training gans,” in Proc. Advances Neural Inf. Process. Syst. , 2016, pp. 2234–2242
2016
Earlier work this paper cites.
S. Reed, Z. Akata, X. Yan, L. Logeswaran, B. Schiele, and H. Lee, “Generative adversarial text to image synthesis,” in Proc. Int. Conf. Mach. Learn. , 2016, pp. 1060–1069
2016
Earlier work this paper cites.
L. Karacan, Z. Akata, A. Erdem, and E. Erdem, “Learning to generate images of outdoor scenes from attributes and semantic layouts,” arXiv preprint arXiv:1612.00215 , 2016
Original
2016
Earlier work this paper cites.
S. Reed, A. van den Oord, N. Kalchbrenner, V. Bapst, M. Botvinick, and N. de Freitas, “Generating interpretable images with controllable structure,” 2016
2016
Earlier work this paper cites.
S. Reed, Z. Akata, S. Mohan, S. Tenka, B. Schiele, and H. Lee, “Learning what and where to draw,” in Proc. Advances Neural Inf. Process. Syst. , 2016, pp. 217–225
2016
Earlier work this paper cites.
L. Theis, A. van den Oord, and M. Bethge, “A note on the evaluation of generative models,” in Int. Conf. Learn. Representations , 2016
2016
Earlier work this paper cites.
P. Anderson, B. Fernando, M. Johnson, and S. Gould, “Spice: Semantic propositional image caption evaluation,” in Proc. Europ. Conf. Comput. Vis. Springer, 2016, pp. 382–398
2016
Earlier work this paper cites.
O. Vinyals, A. Toshev, S. Bengio, and D. Erhan, “Show and tell: Lessons learned from the 2015 mscoco image captioning challenge,” IEEE Trans. Pattern Anal. Mach. Intell. , vol. 39, no. 4, pp. 652–663, 2016
2016
Earlier work this paper cites.
M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter, “Gans trained by a two time-scale update rule converge to a local nash equilibrium,” in Proc. Advances Neural Inf. Process. Syst. , 2017, pp. 6626–6637
2017
Earlier work this paper cites.
H. Zhang, T. Xu, H. Li, S. Zhang, X. Huang, X. Wang, and D. Metaxas, “Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks,” in Proc. IEEE Int. Conf. Comput. Vis. , 2017, pp. 5907–5915
2017
Earlier work this paper cites.
A. Raj, C. Ham, H. Alamri, V. Cartillier, S. Lee, and J. Hays, “Compositional generation of images,” in Proc. Advances Neural Inf. Process. Syst. ViGIL , 2017
2017
Earlier work this paper cites.
M. Kilickaya, A. Erdem, N. Ikizler-Cinbis, and E. Erdem, “Re-evaluating automatic metrics for image captioning,” in Proc. Conf. Europ. Chapter of the Assoc. for Comput. Linguistics , 2017, pp. 199–209
2017
Earlier work this paper cites.
Q. Chen and V. Koltun, “Photographic image synthesis with cascaded refinement networks,” in Proc. IEEE Int. Conf. Comput. Vis. , 2017, pp. 1520–1529
2017
Earlier work this paper cites.
J. Johnson, A. Gupta, and L. Fei-Fei, “Image generation from scene graphs,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. , 2018, pp. 1219–1228
2018
Earlier work this paper cites.
T. Xu, P. Zhang, Q. Huang, H. Zhang, Z. Gan, X. Huang, and X. He, “Attngan: Fine-grained text to image generation with attentional generative adversarial networks,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. , 2018, pp. 1316–1324
2018
Earlier work this paper cites.
J. Redmon and A. Farhadi, “Yolov3: An incremental improvement,” arXiv preprint arXiv:1804.02767 , 2018
Original
2018
Earlier work this paper cites.
H. Zhang, T. Xu, H. Li, S. Zhang, X. Wang, X. Huang, and D. Metaxas, “Stackgan++: Realistic image synthesis with stacked generative adversarial networks,” IEEE Trans. Pattern Anal. Mach. Intell. , vol. 41, no. 8, pp. 1947–1962, 2018
2018
Earlier work this paper cites.

Similar

S. Sharma, D. Suhubdy, V. Michalski, S. E. Kahou, and Y. Bengio, “Chatpainter: Improving text to image generation using dialogue,” in Proc. Int. Conf. Learn. Representations Workshop , 2018
2018
Cited alongside, same era.
S. Sah, D. Peri, A. Shringi, C. Zhang, M. Dominguez, A. Savakis, and R. Ptucha, “Semantically invariant text-to-image generation,” in Proc. IEEE Conf. Image Process. , 2018, pp. 3783–3787
2018
Cited alongside, same era.
Z. Zhang, Y. Xie, and L. Yang, “Photographic text-to-image synthesis with a hierarchically-nested adversarial network,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. , 2018, pp. 6199–6208
2018
Cited alongside, same era.
S. Hong, D. Yang, J. Choi, and H. Lee, “Inferring semantic layout for hierarchical text-to-image synthesis,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. , 2018, pp. 7986–7994
2018
Cited alongside, same era.
S. Hong, X. Yan, T. Huang, and H. Lee, “Learning hierarchical semantic image manipulation through structured representations,” in Proc. Advances Neural Inf. Process. Syst. , 2018, pp. 2712–2722
2018
Cited alongside, same era.
D. Lee, M.-Y. Liu, M.-H. Yang, S. Liu, J. Gu, and J. Kautz, “Context-aware synthesis and placement of object instances,” in Proc. Advances Neural Inf. Process. Syst. , 2018, pp. 10 413–10 423
2018
Cited alongside, same era.
Y. Cheng, Z. Gan, Y. Li, J. Liu, and J. Gao, “Sequential attention gan for interactive image editing via dialogue,” arXiv preprint arXiv:1812.08352 , 2018
Original
2018
Cited alongside, same era.
S. Nam, Y. Kim, and S. J. Kim, “Text-adaptive generative adversarial networks: manipulating images with natural language,” in Proc. Advances Neural Inf. Process. Syst. , 2018, pp. 42–51
2018
Cited alongside, same era.
S. Barratt and R. Sharma, “A note on the inception score,” in Proc. Int. Conf. Mach. Learn. Workshop , 2018
2018
Cited alongside, same era.
K. Shmelkov, C. Schmid, and K. Alahari, “How good is my gan?” in Proc. Europ. Conf. Comput. Vis. , 2018, pp. 213–229
2018
Cited alongside, same era.
T.-C. Wang, M.-Y. Liu, J.-Y. Zhu, A. Tao, J. Kautz, and B. Catanzaro, “High-resolution image synthesis and semantic manipulation with conditional gans,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. , 2018, pp. 8798–8807
2018
Cited alongside, same era.
Z. Deng, J. Chen, Y. Fu, and G. Mori, “Probabilistic neural programmed networks for scene generation,” in Proc. Advances Neural Inf. Process. Syst. , 2018, pp. 4028–4038
2018
Cited alongside, same era.

Then

J. Li, J. Yang, A. Hertzmann, J. Zhang, and T. Xu, “Layoutgan: Generating graphic layouts with wireframe discriminators,” Int. Conf. Learn. Representations , 2019
2019
Closest in time.
A. A. Jyothi, T. Durand, J. He, L. Sigal, and G. Mori, “Layoutvae: Stochastic scene layout generation from a label set,” Proc. IEEE Int. Conf. Comput. Vis. , 2019
2019
Closest in time.
B. Li, B. Zhuang, M. Li, and J. Gu, “Seq-sg2sl: Inferring semantic layout from scene graph through sequence to sequence learning,” in Proc. IEEE Int. Conf. Comput. Vis. , 2019
2019
Closest in time.
X. Liu, G. Yin, J. Shao, X. Wang et al. , “Learning to predict layout-to-image conditional convolutions for semantic image synthesis,” in Proc. Advances Neural Inf. Process. Syst. , 2019, pp. 568–578
2019
Closest in time.
B. Zhao, L. Meng, W. Yin, and L. Sigal, “Image generation from layout,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. , 2019, pp. 8584–8593
2019
Closest in time.
W. Huang, R. Y. D. Xu, and I. Oppermann, “Realistic image generation using region-phrase attention,” in Proc. Asian Conf. Mach. Learn. , 2019, pp. 284–299
2019
Closest in time.
W. Sun and T. Wu, “Image synthesis from reconfigurable layout and style,” in Proc. IEEE Int. Conf. Comput. Vis. , 2019
2019
Closest in time.
Y. Li, T. Ma, Y. Bai, N. Duan, S. Wei, and X. Wang, “Pastegan: A semi-parametric method to generate image from scene graph,” in Proc. Advances Neural Inf. Process. Syst. , 2019
2019
Closest in time.
D. M. Vo and A. Sugimoto, “Visual-relation conscious image generation from structured-text,” arXiv preprint arXiv:1908.01741 , 2019
Original
2019
Closest in time.
A. El-Nouby, S. Sharma, H. Schulz, D. Hjelm, L. E. Asri, S. E. Kahou, Y. Bengio, and G. W. Taylor, “Tell, draw, and repeat: Generating and modifying images based on continual linguistic instruction,” in Proc. IEEE Int. Conf. Comput. Vis. , 2019
2019
Closest in time.
Y. Li, Z. Gan, Y. Shen, J. Liu, Y. Cheng, Y. Wu, L. Carin, D. Carlson, and J. Gao, “Storygan: A sequential conditional gan for story visualization,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. , 2019, pp. 6329–6338
2019
Closest in time.
G. Mittal, S. Agrawal, A. Agarwal, S. Mehta, and T. Marwah, “Interactive image generation using scene graphs,” in Proc. Int. Conf. Learn. Representations Workshop , 2019
2019
Closest in time.
X. Zhou, S. Huang, B. Li, Y. Li, J. Li, and Z. Zhang, “Text guided person image synthesis,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. , 2019, pp. 3663–3672
2019
Closest in time.
S. Yu, H. Dong, F. Liang, Y. Mo, C. Wu, and Y. Guo, “Simgan: Photo-realistic semantic image manipulation using generative adversarial networks,” in Proc. IEEE Conf. Image Process. , 2019, pp. 734–738
2019
Closest in time.
A. Borji, “Pros and cons of gan evaluation measures,” Comput. Vis. and Image Understanding , vol. 179, pp. 41–65, 2019
2019
Closest in time.
S. Ravuri and O. Vinyals, “Classification accuracy score for conditional generative models,” in Proc. Advances Neural Inf. Process. Syst. , 2019
2019
Closest in time.
P. S. Madhyastha, J. Wang, and L. Specia, “Vifidel: Evaluating the visual fidelity of image descriptions,” in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , 2019, pp. 6539–6550
2019
Closest in time.
A. Brock, J. Donahue, and K. Simonyan, “Large scale gan training for high fidelity natural image synthesis,” in Int. Conf. Learn. Representations , 2019
2019
Closest in time.
R. Geirhos, P. Rubisch, C. Michaelis, M. Bethge, F. A. Wichmann, and W. Brendel, “Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness,” in Int. Conf. Learn. Representations , 2019
2019
Closest in time.
P. Agarwal, A. Betancourt, V. Panagiotou, and N. Díaz-Rodríguez, “Egoshots, an ego-vision life-logging dataset and semantic fidelity metric to evaluate diversity in image captioning models,” in ICLR Workshop on Machine Learning in Real Life , 2020
2020
Closest in time.
M. Jaderberg, K. Simonyan, A. Zisserman et al. , “Spatial transformer networks,” in Proc. Advances Neural Inf. Process. Syst. , 2015, pp. 2017–2025
2025
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…