Fetching the paper…
Reading the bibliography…
Despite recent progress towards scaling up multimodal vision-language models, these models are still known to struggle on compositional generalization benchmarks such as Winoground.
From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions
Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier · 2014
Earlier work this paper cites.
Neural module networks
Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein · 2016
Earlier work this paper cites.
Decoupled weight decay regularization
Ilya Loshchilov and Frank Hutter · 2017
Earlier work this paper cites.
Aligning linguistic words and visual semantic units for image captioning
Longteng Guo, Jing Liu, Jinhui Tang, Jiangwei Li, Wei Luo, and Hanqing Lu · 2019
Earlier work this paper cites.
From balustrades to pierre vinken: Looking for syntax in transformer self-attentions
David Mareček and Rudolf Rosa · 2019
Earlier work this paper cites.
Pytorch: An imperative style, high-performance deep learning library
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al · 2019
Earlier work this paper cites.
Lxmert: Learning cross-modality encoder representations from transformers
Hao Tan and Mohit Bansal · 2019
Earlier work this paper cites.
Unified visual-semantic embeddings: Bridging vision and language with structured meaning representations
Hao Wu, Jiayuan Mao, Yufeng Zhang, Yuning Jiang, Lei Li, Weiwei Sun, and Wei-Ying Ma · 2019
Earlier work this paper cites.
Hierarchical vision-language alignment for video captioning
Junchao Zhang and Yuxin Peng · 2019
Earlier work this paper cites.
Quantifying attention flow in transformers
Samira Abnar and Willem Zuidema · 2020
Earlier work this paper cites.
Uniter: Universal image-text representation learning
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu · 2020
Earlier work this paper cites.
Improving bert with syntax-aware local attention
Zhongli Li, Qingyu Zhou, Chao Li, Ke Xu, and Yunbo Cao · 2020
Cited alongside, same era.
Syntax-bert: Improving pre-trained transformers with syntax trees
Jiangang Bai, Yujing Wang, Yiren Chen, Yaming Yang, Jing Bai, Jing Yu, and Yunhai Tong · 2021
Cited alongside, same era.
Multimodal pretraining unmasked: A meta-analysis and a unified framework of vision-and-language BERTs
Emanuele Bugliarello, Ryan Cotterell, Naoaki Okazaki, and Desmond Elliott · 2021
Cited alongside, same era.
Rosita: Enhancing vision-and-language semantic alignments via cross-and intra-modal knowledge integration
Yuhao Cui, Zhou Yu, Chunqi Wang, Zhongzhou Zhao, Ji Zhang, Meng Wang, and Jun Yu · 2021
Cited alongside, same era.
Vlgrammar: Grounded grammar induction of vision and language
Yining Hong, Qing Li, Song-Chun Zhu, and Siyuan Huang · 2021
Vinvl: Revisiting visual representations in vision-language models
Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao · 2021
Later among the works it cites.
Vl-interpret: An interactive visualization tool for interpreting vision-language transformers
Estelle Aflalo, Meng Du, Shao-Yen Tseng, Yongfei Liu, Chenfei Wu, Nan Duan, and Vasudev Lal · 2022
Closest in time.
Attention as grounding: Exploring textual and cross-modal attention on entities and relations in language-and-vision transformer
Nikolai Ilinykh and Simon Dobnik · 2022
Closest in time.
Single-stream multi-level alignment for vision-language pretraining
Zaid Khan, Vijay Kumar BG, Xiang Yu, Samuel Schulter, Manmohan Chandraker, and Yun Fu · 2022
Closest in time.
Cross-modal alignment learning of vision-language conceptual systems
Taehyeong Kim, Hyeonseop Song, and Byoung-Tak Zhang · 2022
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Cited alongside, same era.
Learning transferable visual models from natural language supervision
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al · 2021
Cited alongside, same era.
Learning relation alignment for calibrated cross-modal retrieval
Shuhuai Ren, Junyang Lin, Guangxiang Zhao, Rui Men, An Yang, Jingren Zhou, Xu Sun, and Hongxia Yang · 2021
Cited alongside, same era.
Unsupervised vision-language grammar induction with shared structure modeling
Bo Wan, Wenjuan Han, Zilong Zheng, and Tinne Tuytelaars · 2021
Cited alongside, same era.
Probing inter-modality: Visual parsing with self-attention for vision-and-language pre-training
Hongwei Xue, Yupan Huang, Bei Liu, Houwen Peng, Jianlong Fu, Houqiang Li, and Jiebo Luo · 2021
Cited alongside, same era.
Ernie-vil: Knowledge enhanced vision-language representations through scene graphs
Fei Yu, Jiji Tang, Weichong Yin, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang · 2021
Cited alongside, same era.
Vqa-gnn: Reasoning with multimodal semantic graph for visual question answering
Yanan Wang, Michihiro Yasunaga, Hongyu Ren, Shinya Wada, and Jure Leskovec
Cited in the paper.
Sgeitl: Scene graph enhanced image-text learning for visual commonsense reasoning
Zhecan Wang, Haoxuan You, Liunian Harold Li, Alireza Zareian, Suji Park, Yiqing Liang, Kai-Wei Chang, and Shih-Fu Chang
Cited in the paper.
Closest in time.
Paul Pu Liang, Amir Zadeh, and Louis-Philippe Morency · 2022
Closest in time.
Finding structural knowledge in multimodal-bert
Victor Milewski, Miryam de Lhoneux, and Marie Francine Moens · 2022
Closest in time.
Winoground: Probing vision and language models for visio-linguistic compositionality
Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross · 2022
Closest in time.
When and why vision-language models behave like bag-of-words models, and what to do about it?
Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou · 2022
Closest in time.
Improve MT for search with selected translation memory using search signals
Bryan Zhang · 2022
Closest in time.