2020

Deconfounded Image Captioning: A Causal Retrospect

Yang, Xu, Zhang, Hanwang, Cai, Jianfei

Understand

Dataset bias in vision-language tasks is becoming one of the main problems which hinders the progress of our community.

Existing solutions lack a principled analysis about why modern image captioners easily collapse into dataset bias.
In this paper, we present a novel perspective: Deconfounded Image Captioning (DIC), to find out the answer of this question, then retrospect modern neural image captioners, and finally propose a DIC framework: DICv1.0 to alleviate the negative effects brought by dataset bias.
DIC is based on causal inference, whose two principles: the backdoor and front-door adjustments, help us review previous studies and design new effective models.

Built on

J. Johnson, B. Hariharan, L. van der Maaten, L. Fei-Fei, C. L. Zitnick, and R. Girshick, “Clevr: A diagnostic dataset for compositional language and elementary visual reasoning,” in Computer Vision and Pattern Recognition (CVPR), 2017 IEEE Conference on . IEEE, 2017, pp. 1988–1997
1997
Earlier work this paper cites.
J. Pearl, Causality: models, reasoning and inference . Springer, 2000, vol. 29
2000
Earlier work this paper cites.
W. J. Reed, “The pareto, zipf and other power laws,” Economics letters , vol. 74, no. 1, pp. 15–19, 2001
2001
Earlier work this paper cites.
J. Atwood and D. Towsley, “Diffusion-convolutional neural networks,” in Advances in neural information processing systems , 2016, pp. 1993–2001
2001
Earlier work this paper cites.
K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a method for automatic evaluation of machine translation,” in Proceedings of the 40th annual meeting on association for computational linguistics . Association for Computational Linguistics, 2002, pp. 311–318
2002
Earlier work this paper cites.
H. Liu and P. Singh, “Conceptnet—a practical commonsense reasoning tool-kit,” BT technology journal , vol. 22, no. 4, pp. 211–226, 2004
2004
Earlier work this paper cites.
C.-Y. Lin, “Rouge: A package for automatic evaluation of summaries,” Text Summarization Branches Out , 2004
2004
Earlier work this paper cites.
D. B. Rubin, “Causal inference using potential outcomes: Design, modeling, decisions,” Journal of the American Statistical Association , vol. 100, no. 469, pp. 322–331, 2005
2005
Earlier work this paper cites.
S. Banerjee and A. Lavie, “Meteor: An automatic metric for mt evaluation with improved correlation with human judgments,” in Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization , 2005, pp. 65–72
2005
Earlier work this paper cites.
A. Micheli, “Neural network for graphs: A contextual constructive approach,” IEEE Transactions on Neural Networks , vol. 20, no. 3, pp. 498–511, 2009
2009
Earlier work this paper cites.
J. Mairal, F. Bach, J. Ponce, and G. Sapiro, “Online dictionary learning for sparse coding,” in Proceedings of the 26th annual international conference on machine learning . ACM, 2009, pp. 689–696
2009
Earlier work this paper cites.
B. Z. Yao, X. Yang, L. Lin, M. W. Lee, and S.-C. Zhu, “I2t: Image parsing to text description,” Proceedings of the IEEE , vol. 98, no. 8, pp. 1485–1508, 2010
2010
Earlier work this paper cites.
G. Kulkarni, V. Premraj, V. Ordonez, S. Dhar, S. Li, Y. Choi, A. C. Berg, and T. L. Berg, “Babytalk: Understanding and generating simple image descriptions,” in CVPR , 2011
2011
Earlier work this paper cites.
J. Gordon and B. Van Durme, “Reporting bias and knowledge extraction,” 2013
2013
Earlier work this paper cites.
S. Kazemzadeh, V. Ordonez, M. Matten, and T. Berg, “Referitgame: Referring to objects in photographs of natural scenes,” in Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP) , 2014, pp. 787–798
2014
Earlier work this paper cites.
T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in European conference on computer vision . Springer, 2014, pp. 740–755
2014
Earlier work this paper cites.
N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Dropout: a simple way to prevent neural networks from overfitting,” The journal of machine learning research , vol. 15, no. 1, pp. 1929–1958, 2014
2014
Earlier work this paper cites.
P. Baldi and P. Sadowski, “The dropout learning algorithm,” Artificial intelligence , vol. 210, pp. 78–122, 2014
2014
Earlier work this paper cites.
S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. Lawrence Zitnick, and D. Parikh, “Vqa: Visual question answering,” in Proceedings of the IEEE international conference on computer vision , 2015, pp. 2425–2433
2015
Earlier work this paper cites.
O. Vinyals, A. Toshev, S. Bengio, and D. Erhan, “Show and tell: A neural image caption generator,” in CVPR , 2015
2015
Earlier work this paper cites.
S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time object detection with region proposal networks,” in Advances in neural information processing systems , 2015, pp. 91–99
2015
Earlier work this paper cites.
X. Chen, H. Fang, T.-Y. Lin, R. Vedantam, S. Gupta, P. Dollár, and C. L. Zitnick, “Microsoft coco captions: Data collection and evaluation server,” arXiv preprint arXiv:1504.00325 , 2015
Original
2015
Earlier work this paper cites.
R. Vedantam, C. Lawrence Zitnick, and D. Parikh, “Cider: Consensus-based image description evaluation,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2015, pp. 4566–4575
2015
Earlier work this paper cites.
D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” 3rd International Conference on Learning Representations , 2015
2015
Earlier work this paper cites.
A. Karpathy and L. Fei-Fei, “Deep visual-semantic alignments for generating image descriptions,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2015, pp. 3128–3137
2015
Earlier work this paper cites.
I. Misra, C. Lawrence Zitnick, M. Mitchell, and R. Girshick, “Seeing through the human reporting bias: Visual classifiers from noisy human-centric labels,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2016, pp. 2930–2939
2016
Earlier work this paper cites.
T. Bolukbasi, K.-W. Chang, J. Y. Zou, V. Saligrama, and A. T. Kalai, “Man is to computer programmer as woman is to homemaker? debiasing word embeddings,” in Advances in neural information processing systems , 2016, pp. 4349–4357
2016
Earlier work this paper cites.
J. Pearl, M. Glymour, and N. P. Jewell, Causal inference in statistics: A primer . John Wiley & Sons, 2016
2016
Earlier work this paper cites.
J. Mao, J. Huang, A. Toshev, O. Camburu, A. L. Yuille, and K. Murphy, “Generation and comprehension of unambiguous object descriptions,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2016, pp. 11–20
2016
Earlier work this paper cites.
Q. You, H. Jin, Z. Wang, C. Fang, and J. Luo, “Image captioning with semantic attention,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2016, pp. 4651–4659
2016
Earlier work this paper cites.

Similar

C. Lu, R. Krishna, M. Bernstein, and L. Fei-Fei, “Visual relationship detection with language priors,” in European conference on computer vision . Springer, 2016, pp. 852–869
2016
Cited alongside, same era.
A. Miller, A. Fisch, J. Dodge, A.-H. Karimi, A. Bordes, and J. Weston, “Key-value memory networks for directly reading documents,” Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing , 2016
2016
Cited alongside, same era.
P. Anderson, B. Fernando, M. Johnson, and S. Gould, “Spice: Semantic propositional image caption evaluation,” in European Conference on Computer Vision . Springer, 2016, pp. 382–398
2016
Cited alongside, same era.
Y. Goyal, T. Khot, D. Summers-Stay, D. Batra, and D. Parikh, “Making the v in vqa matter: Elevating the role of image understanding in visual question answering,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2017, pp. 6904–6913
2017
Cited alongside, same era.
A. Caliskan, J. J. Bryson, and A. Narayanan, “Semantics derived automatically from language corpora contain human-like biases,” Science , vol. 356, no. 6334, pp. 183–186, 2017
2017
Cited alongside, same era.
A. Das, S. Kottur, K. Gupta, A. Singh, D. Yadav, J. M. Moura, D. Parikh, and D. Batra, “Visual dialog,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2017, pp. 326–335
2017
Cited alongside, same era.
R. Krishna, Y. Zhu, O. Groth, J. Johnson, K. Hata, J. Kravitz, S. Chen, Y. Kalantidis, L.-J. Li, D. A. Shamma et al. , “Visual genome: Connecting language and vision using crowdsourced dense image annotations,” International Journal of Computer Vision , vol. 123, no. 1, pp. 32–73, 2017
2017
Cited alongside, same era.
J. Lu, C. Xiong, D. Parikh, and R. Socher, “Knowing when to look: Adaptive attention via a visual sentinel for image captioning,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , vol. 6, 2017, p. 2
2017
Cited alongside, same era.
T. Yao, Y. Pan, Y. Li, Z. Qiu, and T. Mei, “Boosting image captioning with attributes,” in IEEE International Conference on Computer Vision, ICCV , 2017, pp. 22–29
2017
Cited alongside, same era.
D. Teney, L. Liu, and A. van den Hengel, “Graph-structured representations for visual question answering,” in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) . IEEE, 2017, pp. 3233–3241
2017
Cited alongside, same era.
Y. N. Dauphin, A. Fan, M. Auli, and D. Grangier, “Language modeling with gated convolutional networks,” in Proceedings of the 34th International Conference on Machine Learning-Volume 70 . JMLR. org, 2017, pp. 933–941
2017
Cited alongside, same era.
S. J. Rennie, E. Marcheret, Y. Mroueh, J. Ross, and V. Goel, “Self-critical sequence training for image captioning,” in CVPR , vol. 1, 2017, p. 3
2017
Cited alongside, same era.

Then

L. Huang, W. Wang, J. Chen, and X.-Y. Wei, “Attention on attention for image captioning,” in IEEE/CVF International Conference on Computer Vision , 2019
2019
Later among the works it cites.
T. Yao, Y. Pan, Y. Li, and T. Mei, “Hierarchy parsing for image captioning,” in Proceedings of the IEEE International Conference on Computer Vision , 2019, pp. 2621–2629
2019
Later among the works it cites.
J. Lu, D. Batra, D. Parikh, and S. Lee, “Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks,” Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019 , 2019
2019
Later among the works it cites.
Z.-J. Zha, D. Liu, H. Zhang, Y. Zhang, and F. Wu, “Context-aware visual policy network for fine-grained image captioning,” IEEE transactions on pattern analysis and machine intelligence , 2019
2019
Later among the works it cites.
X. Yang, H. Zhang, and J. Cai, “Learning to collocate neural modules for image captioning,” IEEE/CVF International Conference on Computer Vision , 2019
2019
Later among the works it cites.
X. Yang, K. Tang, H. Zhang, and J. Cai, “Auto-encoding scene graphs for image captioning,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2019, pp. 10 685–10 694
2019
Later among the works it cites.
L. Li, Z. Gan, Y. Cheng, and J. Liu, “Relation-aware graph attention network for visual question answering,” in Proceedings of the IEEE International Conference on Computer Vision , 2019, pp. 10 313–10 322
2019
Later among the works it cites.
D. Liu, H. Zhang, F. Wu, and Z.-J. Zha, “Learning to assemble neural module tree networks for visual grounding,” in Proceedings of the IEEE International Conference on Computer Vision , 2019, pp. 4673–4682
2019
Later among the works it cites.
R. Suter, D. Miladinovic, B. Schölkopf, and S. Bauer, “Robustly disentangled causal mechanisms: Validating deep representations for interventional robustness,” in International Conference on Machine Learning . PMLR, 2019, pp. 6056–6065
2019
Later among the works it cites.
Y. Qin, J. Du, Y. Zhang, and H. Lu, “Look back and predict forward in image captioning,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2019, pp. 8367–8375
2019
Later among the works it cites.
R. Geirhos, J.-H. Jacobsen, C. Michaelis, R. Zemel, W. Brendel, M. Bethge, and F. A. Wichmann, “Shortcut learning in deep neural networks,” Nature Machine Intelligence , vol. 2, no. 11, pp. 665–673, 2020
2020
Closest in time.
L. Zhou, H. Palangi, L. Zhang, H. Hu, J. J. Corso, and J. Gao, “Unified vision-language pre-training for image captioning and vqa,” The Thirty-Fourth AAAI Conference on Artificial Intelligence , 2020
2020
Closest in time.
M. Cornia, M. Stefanini, L. Baraldi, and R. Cucchiara, “Meshed-memory transformer for image captioning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2020, pp. 10 578–10 587
2020
Closest in time.
Y. Zhou, M. Wang, D. Liu, Z. Hu, and H. Zhang, “More grounded image captioning by distilling image-text matching model,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2020, pp. 4777–4786
2020
Closest in time.
S. Chen, Q. Jin, P. Wang, and Q. Wu, “Say as you wish: Fine-grained control of image caption generation with abstract scene graphs,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2020, pp. 9962–9971
2020
Closest in time.
X. Yang, H. Zhang, and J. Cai, “Auto-encoding and distilling scene graphs for image captioning,” IEEE Transactions on Pattern Analysis and Machine Intelligence , 2020
2020
Closest in time.
C. Jing, Y. Wu, M. Pei, Y. Hu, Y. Jia, and Q. Wu, “Visual-semantic graph matching for visual grounding,” in Proceedings of the 28th ACM International Conference on Multimedia , 2020, pp. 4041–4050
2020
Closest in time.
J. Yu, X. Jiang, Z. Qin, W. Zhang, Y. Hu, and Q. Wu, “Learning dual encoding model for adaptive visual understanding in visual dialogue,” IEEE Transactions on Image Processing , vol. 30, pp. 220–233, 2020
2020
Closest in time.
E. Abbasnejad, D. Teney, A. Parvaneh, J. Shi, and A. v. d. Hengel, “Counterfactual vision and language learning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2020, pp. 10 044–10 054
2020
Closest in time.
V. Veitch, D. Sridhar, and D. Blei, “Adapting text embeddings for causal inference,” in Conference on Uncertainty in Artificial Intelligence . PMLR, 2020, pp. 919–928
2020
Closest in time.
K. Tang, Y. Niu, J. Huang, J. Shi, and H. Zhang, “Unbiased scene graph generation from biased training,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2020, pp. 3716–3725
2020
Closest in time.
T. Wang, J. Huang, H. Zhang, and Q. Sun, “Visual commonsense r-cnn,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2020, pp. 10 760–10 770
2020
Closest in time.
J. Qi, Y. Niu, J. Huang, and H. Zhang, “Two causal principles for improving visual dialog,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2020, pp. 10 860–10 869
2020
Closest in time.
Z. Li, Q. Tran, L. Mai, Z. Lin, and A. L. Yuille, “Context-aware group captioning via self-attention and contrastive features,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2020, pp. 3440–3450
2020
Closest in time.
Y. Niu, K. Tang, H. Zhang, Z. Lu, X.-S. Hua, and J.-R. Wen, “Counterfactual vqa: A cause-effect look at language bias,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2021, pp. 12 700–12 710
2021
Closest in time.
X. Yang, H. Zhang, G. Qi, and J. Cai, “Causal attention for vision-language tasks,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2021, pp. 9847–9857
2021
Closest in time.
K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2016, pp. 770–778
2021
Closest in time.
M. Niepert, M. Ahmed, and K. Kutzkov, “Learning convolutional neural networks for graphs,” in International conference on machine learning , 2016, pp. 2014–2023
2023
Closest in time.
K. Xu, J. Ba, R. Kiros, K. Cho, A. Courville, R. Salakhudinov, R. Zemel, and Y. Bengio, “Show, attend and tell: Neural image caption generation with visual attention,” in International conference on machine learning , 2015, pp. 2048–2057
2057
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…