2020

A Dataset and Baselines for Visual Question Answering on Art

Garcia, Noa, Ye, Chentao, Liu, Zihua et al.

Understand

Answering questions related to art pieces (paintings) is a difficult task, as it implies the understanding of not only the visual information that is shown in the picture, but also the contextual knowledge that is acquired through the study of the history of art.

In this work, we introduce our first attempt towards building a new dataset, coined AQUA (Art QUestion Answering).
The question-answer (QA) pairs are automatically generated using state-of-the-art question generation methods based on paintings and comments provided in an existing art understanding dataset.
The QA pairs are cleansed by crowdsourcing workers with respect to their grammatical correctness, answerability, and answers' correctness.

Built on

Ikeuchi, K., Oishi, T., Takamatsu, J., Sagawa, R., Nakazawa, A., Kurazume, R., Nishino, K., Kamakura, M., Okamoto, Y.: The great Buddha project: Dijitally archiving restoring, and analyzing cultural heritage objects. IJCV pp. 189–208 (2007)
2007
Earlier work this paper cites.
Johnson, C.R., Hendriks, E., Berezhnoy, I.J., Brevdo, E., Hughes, S.M., Daubechies, I., Li, J., Postma, E., Wang, J.Z.: Image processing for artist identification. IEEE Signal Processing Magazine 25
2008
Earlier work this paper cites.
Everingham, M., Van Gool, L., Williams, C.K.I., Winn, J., Zisserman, A.: The Pascal Visual Object Classes (VOC) challenge. IJCV 88
2010
Earlier work this paper cites.
Heilman, M., Smith, N.A.: Good question! Statistical ranking for question generation. In: NAACL (2010)
2010
Earlier work this paper cites.
Shamir, L., Macura, T., Orlov, N., Eckley, D.M., Goldberg, I.G.: Impressionism, expressionism, surrealism: Automated recognition of painters and schools of art. ACM Transactions on Applied Perception (2010)
2010
Earlier work this paper cites.
Carneiro, G., da Silva, N.P., Del Bue, A., Costeira, J.P.: Artistic image classification: An analysis on the printart database. In: ECCV (2012)
2012
Earlier work this paper cites.
Crowley, E., Zisserman, A.: The state of the art: Object retrieval in paintings using discriminative regions. In: BMVC (2014)
2014
Earlier work this paper cites.
Malinowski, M., Fritz, M.: A multi-world approach to question answering about real-world scenes based on uncertain input. In: Proc. NIPS (2014)
2014
Earlier work this paper cites.
Mazidi, K., Nielsen, R.D.: Linguistic considerations in automatic question generation. In: ACL (2014)
2014
Earlier work this paper cites.
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C.L., Parikh, D.: VQA: Visual Question Answering. In: ICCV (2015)
2015
Earlier work this paper cites.
Crowley, E.J., Parkhi, O.M., Zisserman, A.: Face painting: querying art with photos. In: BMVC (2015)
2015
Earlier work this paper cites.
Labutov, I., Basu, S., Vanderwende, L.: Deep questions without deep understanding. In: ACL-IJCNLP (2015)
2015
Earlier work this paper cites.
Plummer, B.A., Wang, L., Cervantes, C.M., Caicedo, J.C., Hockenmaier, J., Lazebnik, S.: Flickr30k Entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In: ICCV (2015)
2015
Earlier work this paper cites.
Ren, M., Kiros, R., Zemel, R.S.: Exploring models and data for image question answering. In: NeurIPS (2015)
2015
Earlier work this paper cites.
Vinyals, O., Toshev, A., Bengio, S., Erhan, D.: Show and tell: A neural image caption generator. In: CVPR (2015)
2015
Earlier work this paper cites.
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR (2016)
2016
Earlier work this paper cites.
Mostafazadeh, N., Misra, I., Devlin, J., Mitchell, M., He, X., Vanderwende, L.: Generating natural questions about an image. In: ACL (2016)
2016
Earlier work this paper cites.
Tan, W.R., Chan, C.S., Aguirre, H.E., Tanaka, K.: Ceci n’est pas une pipe: A deep convolutional network for fine-art paintings classification. In: ICIP (2016)
2016
Earlier work this paper cites.
Tapaswi, M., Zhu, Y., Stiefelhagen, R., Torralba, A., Urtasun, R., Fidler, S.: MovieQA: Understanding stories in movies through question-answering. In: Proc. CVPR (2016)
2016
Earlier work this paper cites.

Similar

Wu, Q., Wang, P., Shen, C., Dick, A., van den Hengel, A.: Ask me anything: Free-form visual question answering based on knowledge from external sources. In: CVPR (2016)
2016
Cited alongside, same era.
Zhu, Y., Groth, O., Bernstein, M.S., Fei-Fei, L.: Visual7W: Grounded question answering in images. CVPR (2016)
2016
Cited alongside, same era.
Du, X., Shao, J., Cardie, C.: Learning to ask: Neural question generation for reading comprehension. In: ACL (2017)
2017
Cited alongside, same era.
Duan, N., Tang, D., Chen, P., Zhou, M.: Question generation for question answering. In: EMNLP (2017)
2017
Cited alongside, same era.
Goyal, Y., Khot, T., Summers-Stay, D., Batra, D., Parikh, D.: Making the V in VQA matter: Elevating the role of image understanding in visual question answering. In: Proc. CVPR (2017)
2017
Cited alongside, same era.
Jain, U., Zhang, Z., Schwing, A.G.: Creativity: Generating diverse questions using variational autoencoders. In: CVPR (2017)
2017
Cited alongside, same era.
Jang, Y., Song, Y., Yu, Y., Kim, Y., Kim, G.: TGIF-QA: Toward spatio-temporal reasoning in visual question answering. In: Proc. CVPR (2017)
2017
Cited alongside, same era.
Johnson, J., Hariharan, B., van der Maaten, L., Fei-Fei, L., Zitnick, C.L., Girshick, R.: CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning. In: Proc. CVPR (2017)
2017
Cited alongside, same era.
Kim, K.M., Heo, M.O., Choi, S.H., Zhang, B.T.: DeepStory: Video story QA by deep embedded memory networks. In: Proc. IJCAI (2017)
2017
Cited alongside, same era.
Ma, D., Gao, F., Bai, Y., Lou, Y., Wang, S., Huang, T., Duan, L.Y.: From part to whole: Who is behind the painting? In: ACMMM (2017)
2017
Cited alongside, same era.
Maharaj, T., Ballas, N., Rohrbach, A., Courville, A., Pal, C.: A dataset and exploration of models for understanding video data through fill-in-the-blank question-answering. In: CVPR (2017)
2017
Cited alongside, same era.
Mun, J., Hongsuck Seo, P., Jung, I., Han, B.: MarioQA: Answering questions by watching gameplay videos. In: Proc. ICCV (2017)
2017
Cited alongside, same era.

Then

Garcia, N., Vogiatzis, G.: How to read paintings: Semantic art understanding with multi-modal retrieval. In: ECCV Workshops (2018)
2018
Later among the works it cites.
Kim, J.H., Jun, J., Zhang, B.T.: Bilinear attention networks. In: NeurIPS (2018)
2018
Later among the works it cites.
Li, Y., Duan, N., Zhou, B., Chu, X., Ouyang, W., Wang, X., Zhou, M.: Visual question generation as dual task of visual question answering. In: CVPR (2018)
2018
Later among the works it cites.
Misra, I., Girshick, R., Fergus, R., Hebert, M., Gupta, A., van der Maaten, L.: Learning by asking questions. In: CVPR (2018)
2018
Later among the works it cites.
Sun, X., Liu, J., Lyu, Y., He, W., Ma, Y., Wang, S.: Answer-focused and position-aware neural question generation. In: EMNLP (2018)
2018
Later among the works it cites.
Wang, P., Wu, Q., Shen, C., Dick, A., van den Hengel, A.: FVQA: Fact-based visual question answering. TPAMI 40
2018
Later among the works it cites.
Yang, J., Lu, J., Lee, S., Dhruv Batra, D.P.: Visual curiosity: Learning to ask questions to learn visual recognition. In: CoRL (2018)
2018
Later among the works it cites.
Zhao, Y., Ni, X., Ding, Y., Ke, Q.: Paragraph-level neural question generation with maxout pointer and gated self-attention networks. In: EMNLP (2018)
2018
Later among the works it cites.
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: Pre-training of deep bidirectional transformers for language understanding. In: NAACL-HLT (2019)
2019
Later among the works it cites.
Garcia, N., Renoust, B., Nakashima, Y.: Context-aware embeddings for automatic art analysis. In: ICMR (2019)
2019
Later among the works it cites.
Kim, Y., Lee, H., Shin, J., Jung, K.: Improving neural question generation using answer separation. In: AAAI (2019)
2019
Later among the works it cites.
Krishna, R., Bernstein, M., Fei-Fei, L.: Information maximizing visual question generation. In: CVPR (2019)
2019
Later among the works it cites.
Lewis, M., Fan, A.: Generative question answering: Learning to answer the whole question. In: ICLR (2019)
2019
Later among the works it cites.
Marino, K., Rastegari, M., Farhadi, A., Mottaghi, R.: Ok-vqa: A visual question answering benchmark requiring external knowledge. In: CVPR (2019)
2019
Later among the works it cites.
Pan, L., Lei, W., Chua, T., Kan, M.: Recent advances in neural question generation. CoRR abs/1905.08949
Original
2019
Later among the works it cites.
Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R.R., Le, Q.V.: XLNet: Generalized autoregressive pretraining for language understanding. In: NeurIPS (2019)
2019
Later among the works it cites.
Zellers, R., Bisk, Y., Farhadi, A., Choi, Y.: From recognition to cognition: Visual commonsense reasoning. In: Proc. CVPR (2019)
2019
Later among the works it cites.
Garcia, N., Otani, M., Chu, C., Nakashima, Y.: KnowIT VQA: Answering knowledge-based questions about videos. In: AAAI (2020)
2020
Closest in time.
Huckle, N., Garcia, N., Vogiatzis, G.: Demographic influences on contemporary art with unsupervised style embeddings. In: ECCV workshops (2020)
2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…