2019

Answering Questions about Data Visualizations using Efficient Bimodal Fusion

Kafle, Kushal, Shrestha, Robik, Price, Brian et al.

Understand

Chart question answering (CQA) is a newly proposed visual question answering (VQA) task where an algorithm must answer questions about data visualizations, e.g.

bar charts, pie charts, and line graphs.
CQA requires capabilities that natural-image VQA algorithms lack: fine-grained measurements, optical character recognition, and handling out-of-vocabulary words in both questions and answers.
Without modifications, state-of-the-art VQA algorithms perform poorly on this task.

Built on

Revision: Automated classification, analysis and redesign of chart images
M. Savva, N. Kong, A. Chhajta, L. Fei-Fei, M. Agrawala, and J. Heer · 2011
Earlier work this paper cites.
Extraction and interpretation of charts in technical documents
J. S. Kallimani, K. Srinivasa, and R. B. Eswara · 2013
Earlier work this paper cites.
A multi-world approach to question answering about real-world scenes based on uncertain input
M. Malinowski and M. Fritz · 2014
Earlier work this paper cites.
VQA: Visual question answering
S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. L. Zitnick, and D. Parikh · 2015
Earlier work this paper cites.
Exploring models and data for image question answering
M. Ren, R. Kiros, and R. Zemel · 2015
Earlier work this paper cites.
Understanding neural networks through deep visualization
Original
J. Yosinski, J. Clune, A. Nguyen, T. Fuchs, and H. Lipson · 2015
Earlier work this paper cites.
Analyzing the behavior of visual question answering models
A. Agrawal, D. Batra, and D. Parikh · 2016
Earlier work this paper cites.
Deep compositional question answering with neural module networks
J. Andreas, M. Rohrbach, T. Darrell, and D. Klein · 2016
Earlier work this paper cites.
A diagram is worth a dozen images
A. Kembhavi, M. Salvato, E. Kolve, M. Seo, H. Hajishirzi, and A. Farhadi · 2016
Earlier work this paper cites.
Stacked attention networks for image question answering
Z. Yang, X. He, J. Gao, L. Deng, and A. J. Smola · 2016
Earlier work this paper cites.
Visual7w: Grounded question answering in images
Y. Zhu, O. Groth, M. Bernstein, and L. Fei-Fei · 2016
Earlier work this paper cites.
Scatteract: Automated extraction of data from scatter plots
M. Cliche, D. Rosenberg, D. Madeka, and C. Yee · 2017
Earlier work this paper cites.
Multi-scale multi-task fcn for semantic page segmentation and table detection
D. He, S. Cohen, B. Price, D. Kifer, and C. L. Giles · 2017
Earlier work this paper cites.
Densely connected convolutional networks
G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger · 2017
Earlier work this paper cites.

Similar

Tgif-qa: Toward spatio-temporal reasoning in visual question answering
Y. Jang, Y. Song, Y. Yu, Y. Kim, and G. Kim · 2017
Cited alongside, same era.
CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning
J. Johnson, B. Hariharan, L. van der Maaten, L. Fei-Fei, C. L. Zitnick, and R. Girshick · 2017
Cited alongside, same era.
An analysis of visual question answering algorithms
K. Kafle and C. Kanan · 2017
Cited alongside, same era.
Visual question answering: Datasets, algorithms, and future challenges
K. Kafle and C. Kanan · 2017
Cited alongside, same era.
Data augmentation for visual question answering
K. Kafle, M. Yousefhussien, and C. Kanan · 2017
Cited alongside, same era.
FigureQA: An annotated figure dataset for visual reasoning
Original
S. E. Kahou, A. Atkinson, V. Michalski, A. Kadar, A. Trischler, and Y. Bengio · 2017
Cited alongside, same era.
Show, ask, attend, and answer: A strong baseline for visual question answering
Original
V. Kazemi and A. Elqursh · 2017
Cited alongside, same era.
Are you smarter than a sixth grader? textbook question answering for multimodal machine comprehension
A. Kembhavi, M. Seo, D. Schwenk, J. Choi, A. Farhadi, and H. Hajishirzi · 2017
Cited alongside, same era.
Visual Genome: Connecting language and vision using crowdsourced dense image annotations
R. Krishna, Y. Zhu, O. Groth, J. Johnson, K. Hata, J. Kravitz, S. Chen, Y. Kalantidis, L.-J. Li, D. A. Shamma, M. Bernstein, and L. Fei-Fei · 2017
Cited alongside, same era.
Reverse-engineering visualizations: Recovering visual encodings from chart images
J. Poco and J. Heer · 2017
Cited alongside, same era.
A simple neural network module for relational reasoning
A. Santoro, D. Raposo, D. G. Barrett, M. Malinowski, R. Pascanu, P. Battaglia, and T. Lillicrap · 2017
Cited alongside, same era.
A data driven approach for compound figure separation using convolutional neural networks
S. Tsutsui and D. J. Crandall · 2017
Cited alongside, same era.

Then

Bottom-up and top-down attention for image captioning and visual question answering
P. Anderson, X. He, C. Buehler, D. Teney, M. Johnson, S. Gould, and L. Zhang · 2018
Later among the works it cites.
Simple and effective multi-paragraph reading comprehension
C. Clark and M. Gardner · 2018
Later among the works it cites.
CondenseNet: An efficient densenet using learned group convolutions
G. Huang, S. Liu, L. van der Maaten, and K. Q. Weinberger · 2018
Later among the works it cites.
Dvqa: Understanding data visualizations via question answering
K. Kafle, S. Cohen, B. Price, and C. Kanan · 2018
Later among the works it cites.
Bilinear attention networks
J.-H. Kim, J. Jun, and B.-T. Zhang · 2018
Later among the works it cites.
A dataset of clinically generated visual questions and answers about radiology images
J. J. Lau, S. Gayen, A. B. Abacha, and D. Demner-Fushman · 2018
Later among the works it cites.
The visual QA devil in the details: The impact of early fusion and batch norm on clevr
Original
M. Malinowski and C. Doersch · 2018
Later among the works it cites.
FiLM: Visual Reasoning with a General Conditioning Layer
E. Perez, F. Strub, H. De Vries, V. Dumoulin, and A. Courville · 2018
Later among the works it cites.
A question-answering framework for plots using deep learning
Original
R. Reddy, R. Ramesh, A. Deshpande, and M. M. Khapra · 2018
Later among the works it cites.
VQD: Visual query detection in natural scenes
M. Acharya, K. Jariwala, and C. Kanan · 2019
Closest in time.
TallyQA: Answering complex counting questions
M. Acharya, K. Kafle, and C. Kanan · 2019
Closest in time.
Challenges and prospects in vision and language research
Original
K. Kafle, R. Shrestha, and C. Kanan · 2019
Closest in time.
Answer them all! toward universal visual question answering models
R. Shrestha, K. Kafle, and C. Kanan · 2019
Closest in time.
Towards vqa models that can read
A. Singh, V. Natarajan, M. Shah, Y. Jiang, X. Chen, D. Batra, D. Parikh, and M. Rohrbach · 2019
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…