Fetching the paper…

Reading the bibliography…

2020

Generating Natural Questions from Images for Multimodal Assistants

Patel, Alkesh, Bindal, Akanksha, Kotek, Hadas et al.

Original
PDF
Post on X

Understand

Generating natural, diverse, and meaningful questions from images is an essential task for multimodal assistants as it confirms whether they have understood the object and scene in the images properly.

The research in visual question answering (VQA) and visual question generation (VQG) is a great step.
However, this research does not capture questions that a visually-abled person would ask multimodal assistants.
Recently published datasets such as KB-VQA, FVQA, and OK-VQA try to collect questions that look for external knowledge which makes them appropriate for multimodal assistants.

Generating Natural Questions from Images for Multimodal Assistants · Around

Built on

Marino, K., Rastegari, M., Farhadi, A., & Mottaghi, R. (2019). OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge. Retrieved from http://arxiv.org/abs/1906.00067
Original
1906
Earlier work this paper cites.
Marino, K., Rastegari, M., Farhadi, A., & Mottaghi, R. (2019). OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge. Retrieved from http://arxiv.org/abs/1906.00067
Original
1906
Earlier work this paper cites.
Lu, J., Batra, D., Parikh, D., & Lee, S. (2019). ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks. (NeurIPS), 1–11. Retrieved from http://arxiv.org/abs/1908.02265
Original
1908
Earlier work this paper cites.
Li, G., Duan, N., Fang, Y., Gong, M., Jiang, D., & Zhou, M. (2019). Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training. Retrieved from http://arxiv.org/abs/1908.06066
Original
1908
Earlier work this paper cites.
Pennington, J., Socher, R., & Manning, C. D. (2014). GloVe: Global vectors for word representation. EMNLP 2014 - 2014 Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference. https://doi.org/10.3115/v1/d14-1162
2014
Earlier work this paper cites.
Simonyan, K., & Zisserman, A. (2015). Very deep convolutional networks for large-scale image recognition. 3rd International Conference on Learning Representations, ICLR 2015 - Conference Track Proceedings
2015
Earlier work this paper cites.
Gao, H., Mao, J., Zhou, J., Huang, Z., Wang, L., & Xu, W. (2015). Are you talking to a machine? Dataset and methods for multilingual image question answering. Advances in Neural Information Processing Systems
2015
Earlier work this paper cites.
Malinowski, M., Rohrbach, M., & Fritz, M. (2015). Ask your neurons: A neural-based approach to answering questions about images. Proceedings of the IEEE International Conference on Computer Vision. https://doi.org/10.1109/ICCV.2015.9
2015
Earlier work this paper cites.
Ren, M., Kiros, R., & Zemel, R. (2015). Exploring Models and Data for Image Question Answering. Advances in Neural Information Processing Systems. Retrieved from http://arxiv.org/abs/1505.02074
Original
2015
Earlier work this paper cites.

Similar

Vinyals, O., Toshev, A., Bengio, S., & Erhan, D. (2015). Show and tell: A neural image caption generator. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. https://doi.org/10.1109/CVPR.2015.7298935
2015
Cited alongside, same era.
Simonyan, K., & Zisserman, A. (2015). Very deep convolutional networks for large-scale image recognition. 3rd International Conference on Learning Representations, ICLR 2015 - Conference Track Proceedings
2015
Cited alongside, same era.
Mostafazadeh, N., Misra, I., Devlin, J., Mitchell, M., He, X., & Vanderwende, L. (2016). Generating natural questions about an image. 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016 - Long Papers, 3, 1802–1813. https://doi.org/10.18653/v1/p16-1170
2016
Cited alongside, same era.
Wu, Q., Wang, P., Shen, C., Dick, A., & Van Den Hengel, A. (2016). Ask me anything: Free-form visual question answering based on knowledge from external sources. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. https://doi.org/10.1109/CVPR.2016.500

Then

Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017). Densely connected convolutional networks. Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017
2017
Later among the works it cites.
Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Adam, H. (2017). MobileNets: Efficient convolutional neural networks for mobile vision applications ArXiv Preprint ArXiv:1704.04861. https://doi.org/arXiv:1704.04861
Original
2017
Later among the works it cites.
Zhang, S., Qu, L., You, S., Yang, Z., & Zhang, J. (2016). Automatic Generation of Grounded Visual Questions. IJCAI International Joint Conference on Artificial Intelligence. https://doi.org/10.24963/ijcai.2017/592
2017
Later among the works it cites.
Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017). Densely connected convolutional networks. Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017. https://doi.org/10.1109/CVPR.2017.243
2017

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

2016

Cited alongside, same era.

He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. https://doi.org/10.1109/CVPR.2016.90

2016

Cited alongside, same era.

Vijayakumar, A. K., Cogswell, M., Selvaraju, R. R., Sun, Q., Lee, S., Crandall, D., & Batra, D. (2016). Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Models. 1–16. Retrieved from http://arxiv.org/abs/1610.02424

2016

Cited alongside, same era.

Jain, U., Zhang, Z., & Schwing, A. (2017). Creativity: Generating Diverse Questions using Variational Autoencoders. Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017. https://doi.org/10.1109/CVPR.2017.575

2017

Cited alongside, same era.

Wang, P., Wu, Q., Shen, C., Dick, A., & Van Den Hengel, A. (2017). KB-VQA Explicit knowledge-based reasoning for visual question answering. IJCAI International Joint Conference on Artificial Intelligence, 1290–1296. https://doi.org/10.24963/ijcai.2017/179

2017

Cited alongside, same era.

Wang, P., Wu, Q., Shen, C., Hengel, A. van den, & Dick, A. (2016). FVQA: Fact-based Visual Question Answering. IEEE Transactions on Pattern Analysis and Machine Intelligence. https://doi.org/10.1109/TPAMI.2017.2754246

2017

Cited alongside, same era.

Later among the works it cites.

Agrawal, A., Lu, J., Antol, S., Mitchell, M., Zitnick, C. L., Parikh, D., & Batra, D. (2017). VQA: Visual Question Answering: www.visualqa.org. International Journal of Computer Vision. https://doi.org/10.1007/s11263-016-0966-6

2017

Later among the works it cites.

Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. ArXiv:1810.04805 [Cs]. https://doi.org/arXiv:1811.03600v2

2018

Later among the works it cites.

Narasimhan, M., Lazebnik, S., & Schwing, A. G. (2018). Out of the box: Reasoning with graph convolution nets for factual visual question answering. Advances in Neural Information Processing Systems

2018

Later among the works it cites.

Shahbaz, M., Suresh, L., Rexford, J., Feamster, N., Rottenstreich, O., & Hira, M. (2019). Elmo. https://doi.org/10.1145/3341302.3342066

2019

Later among the works it cites.

Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, J. D. (2020). VL-BERT: Pre-training of Generic Visual-Linguistic Representations. 1–16

2020

Closest in time.