2017

MUTAN: Multimodal Tucker Fusion for Visual Question Answering

Ben-younes, Hedi, Cadene, Rémi, Cord, Matthieu et al.

Understand

Bilinear models provide an appealing framework for mixing and merging information in Visual Question Answering (VQA) tasks.

They help to learn high level associations between question meaning and visual concepts in the image, but they suffer from huge dimensionality issues.
We introduce MUTAN, a multimodal tensor-based Tucker decomposition to efficiently parametrize bilinear interactions between visual and textual representations.
Additionally to the Tucker framework, we design a low-rank matrix-based decomposition to explicitly constrain the interaction rank.

Built on

Some mathematical notes on three-mode factor analysis
L. R. Tucker · 1966
Earlier work this paper cites.
Finding frequent items in data streams
M. Charikar, K. Chen, and M. Farach-Colton · 2002
Earlier work this paper cites.
Tensor decompositions and applications
T. G. Kolda and B. W. Bader · 2009
Earlier work this paper cites.
On the properties of neural machine translation: Encoder-decoder approaches
K. Cho, B. van Merrienboer, D. Bahdanau, and Y. Bengio · 2014
Earlier work this paper cites.
Adam: A method for stochastic optimization
Original
D. P. Kingma and J. Ba · 2014
Earlier work this paper cites.
Multimodal neural language models
R. Kiros, R. Salakhutdinov, and R. Zemel · 2014
Earlier work this paper cites.
Microsoft coco: Common objects in context
T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick · 2014
Earlier work this paper cites.
Towards a visual turing challenge
M. Malinowski and M. Fritz · 2014
Earlier work this paper cites.
Grounded compositional semantics for finding and describing images with sentences
R. Socher, A. Karpathy, Q. Le, C. Manning, and A. Ng · 2014
Earlier work this paper cites.
VQA: Visual Question Answering
S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. L. Zitnick, and D. Parikh · 2015
Earlier work this paper cites.
Deep residual learning for image recognition
Original
K. He, X. Zhang, S. Ren, and J. Sun · 2015
Earlier work this paper cites.

Similar

Skip-thought vectors
R. Kiros, Y. Zhu, R. Salakhutdinov, R. S. Zemel, A. Torralba, R. Urtasun, and S. Fidler · 2015
Cited alongside, same era.
Bilinear cnn models for fine-grained visual recognition
T.-Y. Lin, A. RoyChowdhury, and S. Maji · 2015
Cited alongside, same era.
Multimodal convolutional neural networks for matching image and sentence
L. Ma, Z. Lu, L. Shang, and H. Li · 2015
Cited alongside, same era.
Exploring models and data for image question answering
M. Ren, R. Kiros, and R. S. Zemel · 2015
Cited alongside, same era.
Show and tell: A neural image caption generator
O. Vinyals, A. Toshev, S. Bengio, and D. Erhan · 2015
Cited alongside, same era.
Show, attend and tell: Neural image caption generation with visual attention
K. Xu, J. Ba, R. Kiros, K. Cho, A. Courville, R. Salakhudinov, R. Zemel, and Y. Bengio · 2015
Cited alongside, same era.
Deep correlation for matching images and text
F. Yan and K. Mikolajczyk · 2015
Cited alongside, same era.
Learning to compose neural networks for question answering
J. Andreas, M. Rohrbach, T. Darrell, and D. Klein · 2016
Cited alongside, same era.
Multimodal compact bilinear pooling for visual question answering and visual grounding
Original
A. Fukui, D. H. Park, D. Yang, A. Rohrbach, T. Darrell, and M. Rohrbach · 2016
Cited alongside, same era.
Multimodal Residual Learning for Visual QA
J.-H. Kim, S.-W. Lee, D. Kwak, M.-O. Heo, J. Kim, J.-W. Ha, and B.-T. Zhang · 2016
Cited alongside, same era.

Then

Visual genome: Connecting language and vision using crowdsourced dense image annotations
R. Krishna, Y. Zhu, O. Groth, J. Johnson, K. Hata, J. Kravitz, S. Chen, Y. Kalantidis, L.-J. Li, D. A. Shamma, M. Bernstein, and L. Fei-Fei · 2016
Later among the works it cites.
Visual question answering with question representation update (qru)
R. Li and J. Jia · 2016
Later among the works it cites.
Hierarchical question-image co-attention for visual question answering
J. Lu, J. Yang, D. Batra, and D. Parikh · 2016
Later among the works it cites.
Ask your neurons: A deep learning approach to visual question answering
Original
M. Malinowski, M. Rohrbach, and M. Fritz · 2016
Later among the works it cites.
Where to look: Focus regions for visual question answering
K. J. Shih, S. Singh, and D. Hoiem · 2016
Later among the works it cites.
Ask me anything: free-form visual question answering based on knowledge from external sources
Q. Wu, P. Wang, C. Shen, A. Dick, and A. van den Hengel · 2016
Later among the works it cites.
Ask, attend and answer: Exploring question-guided spatial attention for visual question answering
H. Xu and K. Saenko · 2016
Later among the works it cites.
Stacked attention networks for image question answering
Z. Yang, X. He, J. Gao, L. Deng, and A. J. Smola · 2016
Later among the works it cites.
Visual7W: Grounded Question Answering in Images
Y. Zhu, O. Groth, M. Bernstein, and L. Fei-Fei · 2016
Later among the works it cites.
Hadamard Product for Low-rank Bilinear Pooling
J.-H. Kim, K.-W. On, J. Kim, J.-W. Ha, and B.-T. Zhang · 2017
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…