What Do VLMs NOTICE? A Mechanistic Interpretability Pipeline for Gaussian-Noise-free Text-Image Corruption and Evaluation

Assessing bert’s syntactic abilities

Original

Yoav Goldberg. 2019 · 1901

Earlier work this paper cites.

Visualbert: A simple and performant baseline for vision and language

Original

Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. 2019 · 1908

Earlier work this paper cites.

Lxmert: Learning cross-modality encoder representations from transformers

Original

Hao Tan and Mohit Bansal. 2019 · 1908

Earlier work this paper cites.

Attention is not not explanation

Original

Sarah Wiegreffe and Yuval Pinter. 2019 · 1908

Earlier work this paper cites.

Analyzing individual neurons in pre-trained language models

Original

Nadir Durrani, Hassan Sajjad, Fahim Dalvi, and Yonatan Belinkov. 2020 · 2010

Earlier work this paper cites.

Denoising diffusion implicit models

Original

Jiaming Song, Chenlin Meng, and Stefano Ermon. 2022 · 2010

Earlier work this paper cites.

The caltech-ucsd birds-200-2011 dataset

Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. 2011 · 2011

Earlier work this paper cites.

Seeing past words: Testing the cross-modal capabilities of pretrained v&l models on counting tasks

Original

Letitia Parcalabescu, Albert Gatt, Anette Frank, and Iacer Calixto. 2020 · 2012

Earlier work this paper cites.

Challenges in representation learning: A report on three machine learning contests

Ian J Goodfellow, Dumitru Erhan, Pierre Luc Carrier, Aaron Courville, Mehdi Mirza, Ben Hamner, Will Cukierski, Yichuan Tang, David Thaler, Dong-Hyun Lee, et al. 2013 · 2013

Earlier work this paper cites.

Efficient estimation of word representations in vector space

Original

Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013 · 2013

Earlier work this paper cites.

Discovering states and transformations in image collections

Phillip Isola, Joseph J Lim, and Edward H Adelson. 2015 · 2015

Earlier work this paper cites.

Sentiment of emojis

Petra Kralj Novak, Jasmina Smailović, Borut Sluban, and Igor Mozetič. 2015 · 2015

Earlier work this paper cites.

Large-scale fashion (deepfashion) database

Ziwei Liu, Ping Luo, Shi Qiu, Xiaogang Wang, and Xiaoou Tang. 2016 · 2016

Earlier work this paper cites.

" why should i trust you?" explaining the predictions of any classifier

Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016 · 2016

Earlier work this paper cites.

Clevr: A diagnostic dataset for compositional language and elementary visual reasoning

Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. 2017 · 2017

Earlier work this paper cites.

Shapeworld-a new test methodology for multimodal language understanding

Original

Alexander Kuhnle and Ann Copestake. 2017 · 2017

Earlier work this paper cites.

What you can cram into a single $&!#* vector: Probing sentence embeddings for linguistic properties

Alexis Conneau, German Kruszewski, Guillaume Lample, Loïc Barrault, and Marco Baroni. 2018 · 2018

Earlier work this paper cites.

A structural probe for finding syntax in word representations

John Hewitt and Christopher D Manning. 2019 · 2019

Earlier work this paper cites.

What does bert learn about the structure of language?

Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. 2019 · 2019

Earlier work this paper cites.

Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks

Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019 · 2019

Earlier work this paper cites.

Behind the scene: Revealing the secrets of pre-trained vision-and-language models

Jize Cao, Zhe Gan, Yu Cheng, Licheng Yu, Yen-Chun Chen, and Jingjing Liu. 2020 · 2020

Earlier work this paper cites.

Does my multimodal model learn cross-modal interactions? it’s harder to tell than you might think!

Jack Hessel and Lillian Lee. 2020 · 2020

Cited alongside, same era.

Lsun-stanford car dataset: enhancing large-scale car image datasets using deep learning for usage in gan training

Tin Kramberger and Božidar Potočnik. 2020 · 2020

Cited alongside, same era.

Oscar: Object-semantics aligned pre-training for vision-language tasks

Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. 2020 · 2020

Cited alongside, same era.

The explanation game: Explaining machine learning models using shapley values

Luke Merrick and Ankur Taly. 2020 · 2020

Cited alongside, same era.

Vision-and-language or vision-for-language? on cross-modal influence in multimodal transformers

Original

Stella Frank, Emanuele Bugliarello, and Desmond Elliott. 2021 · 2021

Cited alongside, same era.

A toy model of universality: Reverse engineering how networks learn group operations

Original

Bilal Chughtai, Lawrence Chan, and Neel Nanda. 2023 · 2023

Later among the works it cites.

Successor heads: Recurring, interpretable attention heads in the wild

Original

Rhys Gould, Euan Ong, George Ogden, and Arthur Conmy. 2023 · 2023

Later among the works it cites.

Llms as visual explainers: Advancing image classification with evolving visual descriptions

Original

Songhao Han, Le Zhuo, Yue Liao, and Si Liu. 2023 · 2023

Later among the works it cites.

How does gpt-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model

Original

Michael Hanna, Ollie Liu, and Alexandre Variengien. 2023 · 2023

Later among the works it cites.

Multiviz: Towards visualizing and understanding multimodal models