2020

AVLnet: Learning Audio-Visual Language Representations from Instructional Videos

Rouditchenko, Andrew, Boggust, Angie, Harwath, David et al.

Understand

Current methods for learning visually grounded language from videos often rely on text annotation, such as human generated captions or machine generated automatic speech recognition (ASR) transcripts.

In this work, we introduce the Audio-Video Language Network (AVLnet), a self-supervised network that learns a shared audio-visual embedding space directly from raw video inputs.
To circumvent the need for text annotation, we learn audio-visual representations from randomly segmented video clips and their raw audio waveforms.
We train AVLnet on HowTo100M, a large corpus of publicly available instructional videos, and evaluate on image retrieval and video retrieval tasks, achieving state-of-the-art performance.

Built on

J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in
2009
Earlier work this paper cites.
T. Mikolov, K. Chen, G. Corrado, and J. Dean, “Efficient estimation of word representations in vector space,”
Original
2013
Earlier work this paper cites.
G. Synnaeve, M. Versteegh, and E. Dupoux, “Learning words from images and speech,” in
2014
Earlier work this paper cites.
B. Zhou, A. Lapedriza, J. Xiao, A. Torralba, and A. Oliva, “Learning deep features for scene recognition using places database,” in
2014
Earlier work this paper cites.
D. Harwath and J. Glass, “Deep multimodal semantic embeddings for speech and images,” in
2015
Earlier work this paper cites.
D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in
2015
Earlier work this paper cites.
B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba, “Object detectors emerge in deep scene cnns,” in
2015
Earlier work this paper cites.
D. Harwath, A. Torralba, and J. Glass, “Unsupervised learning of spoken language with visual context,” in
2016
Earlier work this paper cites.
Y. Aytar, C. Vondrick, and A. Torralba, “Soundnet: Learning sound representations from unlabeled video,” in
2016
Earlier work this paper cites.
J. Xu, T. Mei, T. Yao, and Y. Rui, “Msr-vtt: A large video description dataset for bridging video and language,” in
2016
Earlier work this paper cites.
A. Owens, J. Wu, J. H. McDermott, W. T. Freeman, and A. Torralba, “Ambient sound provides supervision for visual learning,” in
2016
Earlier work this paper cites.
J.-B. Alayrac, P. Bojanowski, N. Agrawal, J. Sivic, I. Laptev, and S. Lacoste-Julien, “Unsupervised learning from narrated instruction videos,” in
2016
Earlier work this paper cites.
K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in
2016
Earlier work this paper cites.
G. Chrupała, L. Gelderloos, and A. Alishahi, “Representations of language in a model of visually grounded speech signal,” in
2017
Earlier work this paper cites.
R. Arandjelovic and A. Zisserman, “Look, listen and learn,” in
2017
Earlier work this paper cites.
A. Miech, I. Laptev, and J. Sivic, “Learnable pooling with context gating for video classification,” in
2017
Earlier work this paper cites.
A. Rohrbach, A. Torabi, M. Rohrbach, N. Tandon, C. Pal, H. Larochelle, A. Courville, and B. Schiele, “Movie description,” in
2017
Earlier work this paper cites.
W. Havard, L. Besacier, and O. Rosec, “Speech-coco: 600k visually grounded spoken captions aligned to mscoco data set,”
Original
2017
Earlier work this paper cites.
D. Harwath and J. Glass, “Learning word-like units from joint audio-visual analysis,” in
2017
Earlier work this paper cites.
K. Leidal, D. Harwath, and J. Glass, “Learning modality-invariant representations for speech and images,” in
2017
Earlier work this paper cites.
J. Carreira and A. Zisserman, “Quo vadis, action recognition? a new model and the kinetics dataset,” in
2017
Earlier work this paper cites.
D. Harwath, A. Recasens, D. Surís, G. Chuang, A. Torralba, and J. Glass, “Jointly discovering visual objects and spoken words from raw sensory input,” in
2018
Earlier work this paper cites.
——, “Objects that sound,” in
2018
Earlier work this paper cites.
A. Owens and A. A. Efros, “Audio-visual scene analysis with self-supervised multisensory features,” in
2018
Earlier work this paper cites.
B. Korbar, D. Tran, and L. Torresani, “Cooperative learning of audio and video models from self-supervised synchronization,” in
2018
Earlier work this paper cites.

Similar

H. Zhao, C. Gan, A. Rouditchenko, C. Vondrick, J. McDermott, and A. Torralba, “The sound of pixels,” in
2018
Cited alongside, same era.
A. Miech, I. Laptev, and J. Sivic, “Learning a text-video embedding from incomplete and heterogeneous data,”
Original
2018
Cited alongside, same era.
N. C. Mithun, J. Li, F. Metze, and A. K. Roy-Chowdhury, “Learning joint embedding with multimodal cues for cross-modal video-text retrieval,” in
2018
Cited alongside, same era.
Y. Yu, J. Kim, and G. Kim, “A joint sequence fusion model for video question answering and retrieval,” in
2018
Cited alongside, same era.
L. Zhou, C. Xu, and J. J. Corso, “Towards automatic learning of procedures from web instructional videos,” in
2018
Cited alongside, same era.
R. Sanabria, O. Caglayan, S. Palaskar, D. Elliott, L. Barrault, L. Specia, and F. Metze, “How2: a large-scale dataset for multimodal language understanding,” in
2018
Cited alongside, same era.
H. Kamper and M. Roth, “Visually grounded cross-lingual keyword spotting in speech,”
Original
2018
Cited alongside, same era.
W.-N. Hsu and J. Glass, “Disentangling by partitioning: A representation learning framework for multimodal sensory data,”
Original
2018
Cited alongside, same era.
R. Gao, R. Feris, and K. Grauman, “Learning to separate object sounds by watching unlabeled video,” in
2018
Cited alongside, same era.
P. Morgado, N. Nvasconcelos, T. Langlois, and O. Wang, “Self-supervised generation of spatial audio for 360 video,” in
2018
Cited alongside, same era.
K. Hara, H. Kataoka, and Y. Satoh, “Can spatiotemporal 3d cnns retrace the history of 2d cnns and imagenet?” in
2018
Cited alongside, same era.
A. v. d. Oord, Y. Li, and O. Vinyals, “Representation learning with contrastive predictive coding,”
Original
2018
Cited alongside, same era.

Then

H. Kamper, A. Anastassiou, and K. Livescu, “Semantic query-by-example speech search using visual grounding,” in
2019
Later among the works it cites.
L. Wang and M. A. Hasegawa-Johnson, “Multimodal word discovery and retrieval with phone sequence and image concepts.” in
2019
Later among the works it cites.
R. Eloff, H. A. Engelbrecht, and H. Kamper, “Multimodal one-shot learning of speech and images,” in
2019
Later among the works it cites.
D. Hu, F. Nie, and X. Li, “Deep multimodal clustering for unsupervised audiovisual learning,” in
2019
Later among the works it cites.
R. Gao and K. Grauman, “2.5d visual sound,” in
2019
Later among the works it cites.
A. Boggust, K. Audhkhasi, D. Joshi, D. Harwath, S. Thomas, R. Feris, D. Gutfreund, Y. Zhang, A. Torralba, M. Picheny, and J. Glass, “Grounding spoken words in unlabeled video,” in
2019
Later among the works it cites.
Y. Tang, D. Ding, Y. Rao, Y. Zheng, D. Zhang, L. Zhao, J. Lu, and J. Zhou, “Coin: A large-scale dataset for comprehensive instructional video analysis,” in
2019
Later among the works it cites.
H. Kuehne, A. Iqbal, A. Richard, and J. Gall, “Mining youtube-a dataset for learning fine-grained action concepts from webly supervised video data,”
Original
2019
Later among the works it cites.
A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga
2019
Later among the works it cites.
——, “Jointly discovering visual objects and spoken words from raw sensory input,” in
2020
Closest in time.
J.-B. Alayrac, A. Recasens, R. Schneider, R. Arandjelović, J. Ramapuram, J. De Fauw, L. Smaira, S. Dieleman, and A. Zisserman, “Self-supervised multimodal versatile networks,” in
2020
Closest in time.
A. Miech, J.-B. Alayrac, L. Smaira, I. Laptev, J. Sivic, and A. Zisserman, “End-to-end learning of visual representations from uncurated instructional videos,” in
2020
Closest in time.
D. Harwath, W.-N. Hsu, and J. Glass, “Learning hierarchical discrete linguistic units from visually-grounded speech,” in
2020
Closest in time.
M. S. Mortazavi, “Speech-image semantic alignment does not depend on any prior classification tasks,” in
2020
Closest in time.
L. Wang and M. Hasegawa-Johnson, “A dnn-hmm-dnn hybrid model for discovering word-like units from spoken captions and image regions,” in
2020
Closest in time.
A.-M. Oncescu, J. F. Henriques, Y. Liu, A. Zisserman, and S. Albanie, “Queryd: A video dataset with high-quality textual and audio narrations,”
Original
2020
Closest in time.
H. Alwassel, D. Mahajan, B. Korbar, L. Torresani, B. Ghanem, and D. Tran, “Self-supervised learning by cross-modal audio-video clustering,” in
2020
Closest in time.
K. Yang, B. Russell, and J. Salamon, “Telling left from right: Learning spatial correspondence of sight and sound,” in
2020
Closest in time.
E. Amrani, R. Ben-Ari, D. Rotman, and A. Bronstein, “Noise estimation using density estimation for self-supervised multimodal learning,”
Original
2020
Closest in time.
T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A simple framework for contrastive learning of visual representations,” in
2020
Closest in time.
R. Sanabria, A. Waters, and J. Baldridge, “Talk, don’t write: A study of direct speech-based image retrieval,”
Original
2021
Closest in time.
L. Wang, X. Wang, M. Hasegawa-Johnson, O. Scharenborg, and N. Dehak, “Align or attend? toward more efficient and accurate spoken word discovery using speech-to-image retrieval,” in
2021
Closest in time.
G. Chrupała, “Visually grounded models of spoken language: A survey of datasets, architectures and evaluation techniques,”
Original
2021
Closest in time.
M. Monfort, S. Jin, A. Liu, D. Harwath, R. Feris, J. Glass, and A. Oliva, “Spoken moments: Learning joint audio-visual representations from video descriptions,” in
2021
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…