2020

Multimodal grid features and cell pointers for Scene Text Visual Question Answering

Gómez, Lluís, Biten, Ali Furkan, Tito, Rubèn et al.

Understand

This paper presents a new model for the task of scene text visual question answering, in which questions about a given image can only be answered by reading and understanding scene text that is present in it.

The proposed model is based on an attention mechanism that attends to multi-modal features conditioned to the question, allowing it to reason jointly about the textual and visual modalities in the scene.
The output weights of this attention module over the grid of multi-modal spatial features are interpreted as the probability that a certain spatial location of the image contains the answer text the to the given question.
Our experiments demonstrate competitive performance in two standard datasets.

Built on

Learning to forget: Continual prediction with lstm
Gers, F.A., Schmidhuber, J., Cummins, F., 1999 · 1999
Earlier work this paper cites.
Glove: Global vectors for word representation, in: EMNLP
Pennington, J., Socher, R., Manning, C.D., 2014 · 2014
Earlier work this paper cites.
Vqa: Visual question answering, in: ICCV
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Lawrence Zitnick, C., Parikh, D., 2015 · 2015
Earlier work this paper cites.
Pointer networks, in: NIPS
Vinyals, O., Fortunato, M., Jaitly, N., 2015 · 2015
Earlier work this paper cites.
Enriching word vectors with subword information
Original
Bojanowski, P., Grave, E., Joulin, A., Mikolov, T., 2016 · 2016
Earlier work this paper cites.
Deep residual learning for image recognition, in: CVPR
He, K., Zhang, X., Ren, S., Sun, J., 2016 · 2016
Earlier work this paper cites.
Multimodal residual learning for visual qa, in: NIPS
Kim, J.H., Lee, S.W., Kwak, D., Heo, M.O., Kim, J., Ha, J.W., Zhang, B.T., 2016 · 2016
Earlier work this paper cites.
Ssd: Single shot multibox detector, in: ECCV
Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.Y., Berg, A.C., 2016 · 2016
Earlier work this paper cites.
Hierarchical question-image co-attention for visual question answering, in: NIPS
Lu, J., Yang, J., Batra, D., Parikh, D., 2016 · 2016
Earlier work this paper cites.
You only look once: Unified, real-time object detection, in: CVPR
Redmon, J., Divvala, S., Girshick, R., Farhadi, A., 2016 · 2016
Earlier work this paper cites.

Similar

Rethinking the inception architecture for computer vision, in: CVPR
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z., 2016 · 2016
Cited alongside, same era.
Stacked attention networks for image question answering, in: CVPR
Yang, Z., He, X., Gao, J., Deng, L., Smola, A., 2016 · 2016
Cited alongside, same era.
Show, ask, attend, and answer: A strong baseline for visual question answering
Original
Kazemi, V., Elqursh, A., 2017 · 2017
Cited alongside, same era.
Dual attention networks for multimodal reasoning and matching, in: CVPR
Nam, H., Ha, J.W., Kim, J., 2017 · 2017
Cited alongside, same era.
Bottom-up and top-down attention for image captioning and visual question answering, in: CVPR
Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S., Zhang, L., 2018 · 2018
Cited alongside, same era.
Rosetta: Large scale system for text detection and recognition in images, in: ACM SIGKDD
Borisyuk, F., Gordo, A., Sivakumar, V., 2018 · 2018
Cited alongside, same era.
E2e-mlt-an unconstrained end-to-end method for multi-language scene text, in: ACCV
Bušta, M., Patel, Y., Matas, J., 2018 · 2018
Cited alongside, same era.
Bert: Pre-training of deep bidirectional transformers for language understanding
Original
Devlin, J., Chang, M.W., Lee, K., Toutanova, K., 2018 · 2018
Cited alongside, same era.
Single shot scene text retrieval, in: ECCV
Gómez, L., Mafla, A., Rusinol, M., Karatzas, D., 2018 · 2018
Cited alongside, same era.
Pythia v0. 1: the winning entry to the vqa challenge 2018
Original
Jiang, Y., Natarajan, V., Chen, X., Rohrbach, M., Batra, D., Parikh, D., 2018 · 2018
Cited alongside, same era.
Scene text visual question answering, in: ICCV
Biten, A.F., Tito, R., Mafla, A., Gomez, L., Rusinol, M., Valveny, E., Jawahar, C., Karatzas, D., 2019b
Cited in the paper.

Then

Fots: Fast oriented text spotting with a unified network, in: CVPR
Liu, X., Liang, D., Yan, S., Chen, D., Qiao, Y., Yan, J., 2018 · 2018
Later among the works it cites.
Yolov3: An incremental improvement
Original
Redmon, J., Farhadi, A., 2018 · 2018
Later among the works it cites.
Beyond bilinear: Generalized multimodal factorized high-order pooling for visual question answering
Yu, Z., Yu, J., Xiang, C., Fan, J., Tao, D., 2018 · 2018
Later among the works it cites.
What is wrong with scene text recognition model comparisons? dataset and model analysis, in: ICCV
Baek, J., Kim, G., Lee, J., Park, S., Han, D., Yun, S., Oh, S.J., Lee, H., 2019 · 2019
Later among the works it cites.
Icdar 2019 competition on scene text visual question answering, in: ICDAR
Biten, A.F., Tito, R., Mafla, A., Gomez, L., Rusinol, M., Mathew, M., Jawahar, C., Valveny, E., Karatzas, D., 2019a · 2019
Later among the works it cites.
Ocr-vqa: Visual question answering by reading text in images, in: ICDAR
Mishra, A., Shekhar, S., Singh, A.K., Chakraborty, A., 2019 · 2019
Later among the works it cites.
Pythia-a platform for vision & language research, in: SysML Workshop, NeurIPS 2019
Singh, A., Natarajan, V., Jiang, Y., Chen, X., Shah, M., Rohrbach, M., Batra, D., Parikh, D., 2018 · 2019
Later among the works it cites.
Towards vqa models that can read, in: CVPR
Singh, A., Natarajan, V., Shah, M., Jiang, Y., Chen, X., Batra, D., Parikh, D., Rohrbach, M., 2019 · 2019
Later among the works it cites.
The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale
Kuznetsova, A., Rom, H., Alldrin, N., Uijlings, J., Krasin, I., Pont-Tuset, J., Kamali, S., Popov, S., Malloci, M., Kolesnikov, A., Duerig, T., Ferrari, V., 2020 · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…