2019

Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speech

Harwath, David, Hsu, Wei-Ning, Glass, James

Understand

In this paper, we present a method for learning discrete linguistic units by incorporating vector quantization layers into neural models of visually grounded speech.

We show that our method is capable of capturing both word-level and sub-word units, depending on how it is configured.
What differentiates this paper from prior work on speech unit learning is the choice of training objective.
Rather than using a reconstruction-based loss, we use a discriminative, multimodal grounding objective which forces the learned units to be useful for semantic image retrieval.

Built on

Learning classification with unlabeled data
Virginia de Sa · 1994
Earlier work this paper cites.
Learning words from sights and sounds: a computational model
Deb Roy and Alex Pentland · 2002
Earlier work this paper cites.
Grounded spoken language acquisition: Experiments in word learning
Deb Roy · 2003
Earlier work this paper cites.
Towards unsupervised pattern discovery in speech
Alex Park and James Glass · 2005
Earlier work this paper cites.
Unsupervised pattern discovery in speech
Alex Park and James Glass · 2008
Earlier work this paper cites.
Unsupervised learning of acoustic sub-word units
Balakrishnan Varadarajan, Sanjeev Khudanpur, and Emmanuel Dupoux · 2008
Earlier work this paper cites.
Unsupervised training of an HMM-based speech recognizer for topic classification
Herb Gish, Man-Hung Siu, Arthur Chan, and William Belfield · 2009
Earlier work this paper cites.
Distance metric learning for large margin nearest neighbor classification
Kilian Q. Weinberger and Lawrence K. Saul · 2009
Earlier work this paper cites.
Unsupervised spoken keyword spotting via segmental dtw on gaussian posteriorgrams
Yaodong Zhang and James Glass · 2009
Earlier work this paper cites.
Toward spoken term discovery at scale with zero resources
Aren Jansen, Kenneth Church, and Hynek Hermansky · 2010
Earlier work this paper cites.
Efficient spoken term discovery using randomized algorithms
Aren Jansen and Benjamin Van Durme · 2011
Earlier work this paper cites.
A nonparametric Bayesian approach to acoustic model discovery
Chia-Ying Lee and James Glass · 2012
Earlier work this paper cites.
Estimating or propagating gradients through stochastic neurons for conditional computation
Original
Yoshua Bengio, Nicholas Léonard, and Aaron Courville · 2013
Earlier work this paper cites.
Evaluating speech features with the minimal-pair ABX task: Analysis of the classical MFC/PLP pipeline
Thomas Schatz, Vijayaditya Peddinti, Francis Bach, Aren Jansen, Hynek Hermansky, and Emmanuel Dupoux · 2013
Earlier work this paper cites.
Adam: A method for stochastic optimization
Diederik P. Kingma and Jimmy Ba · 2014
Earlier work this paper cites.
Unsupervised training of an HMM-based self-organizing unit recognizer with applications to topic classification and keyword discovery
Man-Hung. Siu, Herb Gish, Arthur Chan, William Belfield, and Steve Lowe · 2014
Earlier work this paper cites.
Learning words from images and speech
Gabriel Synnaeve, Maarten Versteegh, and Emmanuel Dupoux · 2014
Earlier work this paper cites.
Learning deep features for scene recognition using places database
Bolei Zhou, Agata Lapedriza, Jianxiong Xiao, Antonio Torralba, and Aude Oliva · 2014
Earlier work this paper cites.
Deep multimodal semantic embeddings for speech and images
David Harwath and James Glass · 2015
Earlier work this paper cites.
Fully unsupervised small-vocabulary speech recognition using a segmental Bayesian model
Herman Kamper, Aren Jansen, and Sharon Goldwater · 2015
Earlier work this paper cites.
Unsupervised lexicon discovery from acoustic input
Chia-Ying Lee, Timothy J. O’Donnell, and James Glass · 2015
Earlier work this paper cites.

Similar

A hybrid dynamic time warping-deep neural network architecture for unsupervised acoustic modeling
R. Thiolliere, E. Dunbar, G. Synnaeve, M. Versteegh, and E. Dupoux · 2015
Cited alongside, same era.
The zero resource speech challenge 2015
Martin Versteegh, Roland Thiolliere, Thomas Schatz, Xuan Nga Cao, Xavier Anguera, Aren Jansen, and Emmanuel Dupoux · 2015
Cited alongside, same era.
Unsupervised learning of spoken language with visual context
David Harwath, Antonio Torralba, and James R. Glass · 2016
Cited alongside, same era.
Deep residual learning for image recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun · 2016
Cited alongside, same era.
Unsupervised word segmentation and lexicon discovery using acoustic word embeddings
Herman Kamper, Aren Jansen, and Sharon Goldwater · 2016
Cited alongside, same era.
Ethnologue: Languages of the World, Nineteenth edition
M. Paul Lewis, Gary F. Simon, and Charles D. Fennig · 2016
Cited alongside, same era.
Variational inference for acoustic unit discovery
Lucas Ondel, Lukás Burget, and Jan Černocký · 2016
Cited alongside, same era.
Encoding of phonology in a recurrent neural model of grounded speech
Afra Alishahi, Marie Barking, and Grzegorz Chrupała · 2017
Cited alongside, same era.
Representations of language in a model of visually grounded speech signal
Grzegorz Chrupała, Lieke Gelderloos, and Afra Alishahi · 2017
Cited alongside, same era.
Analysis of audio-visual features for unsupervised speech recognition
Jennifer Drexler and James Glass · 2017
Cited alongside, same era.
The zero resource speech challenge 2017
Ewan Dunbar, Xuan Nga Cao, Juan Benjumea, Julien Karadayi, Mathieu Bernard, Laurent Besacier, Xavier Anguera, and Emmanuel Dupoux · 2017
Cited alongside, same era.
Learning word-like units from joint audio-visual analysis
David Harwath and James Glass · 2017
Cited alongside, same era.

Then

Linguistic unit discovery from multi-modal inputs in unwritten languages: Summary of the ”Speaking Rosetta” JSALT 2017 workshop
Odette Scharenborg, Laurent Besacier, Alan W. Black, Mark Hasegawa-Johnson, Florian Metze, Graham Neubig, Sebastian Stüker, Pierre Godard, Markus Müller, Lucas Ondel, Shruti Palaskar, Philip Arthur, Francesco Ciannella, Mingxing Du, Elin Larsen, Danny Merkx, Rachid Riad, Liming Wang, and Emmanuel Dupoux · 2018
Later among the works it cites.
Representation learning with contrastive predictive coding
Original
Aäron van den Oord, Yazhe Li, and Oriol Vinyals · 2018
Later among the works it cites.
Towards bilingual lexicon discovery from visually grounded speech audio
Emmanuel Azuh, David Harwath, and James Glass · 2019
Closest in time.
VQVAE with speaker adversarial training, 2019
Suhee Cho, Yeonjung Hong, Yookyunk Shin, and Youngsun Cho · 2019
Closest in time.
Unsupervised speech representation learning using wavenet autoencoders
Jan Chorowski, Ron J. Weiss, Samy Bengio, and Aäron van den Oord · 2019
Closest in time.
Symbolic inductive bias for visually grounded learning of spoken language
Grzegorz Chrupała · 2019
Closest in time.
An unsupervised autoregressive model for speech representation learning
Yu-An Chung, Wei-Ning Hsu, Hao Tang, and James R. Glass · 2019
Closest in time.
The zero resource speech challenge 2019: TTS without T
Ewan Dunbar, Robin Algayres, Julien Karadayi, Mathieu Bernard, Juan Benjumea, Xuan-Nga Cao, Lucie Miskic, Charlotte Dugrain, Lucas Ondel, Alan W. Black, Laurent Besacier, Sakriani Sakti, and Emmanuel Dupoux · 2019
Closest in time.
Combining adversarial training and disentangled speech representation for robust zero-resource subword modeling
Siyuan Feng, Tan Lee, and Zhiyuan Peng · 2019
Closest in time.
Google cloud speech-to-text API
Google · 2019
Closest in time.
Towards visually grounded sub-word speech unit discovery
David Harwath and James Glass · 2019
Closest in time.
Jointly discovering visual objects and spoken words from raw sensory input
David Harwath, Adrià Recasens, Dídac Surís, Galen Chuang, Antonio Torralba, and James Glass · 2019
Closest in time.
Learning from multiview correlations in open-domain videos
Nils Holzenberger, Shruti Palaskar, Pranava Madhyastha, Florian Metze, and Raman Arora · 2019
Closest in time.
Transfer learning from audio-visual grounding to speech recognition
Wei-Ning Hsu, David Harwath, and James Glass · 2019
Closest in time.
Large-scale representation learning from visually grounded untranscribed speech
Gabriel Ilharco, Yuan Zhang, and Jason Baldridge · 2019
Closest in time.
Unsupervised end-to-end learning of discrete linguistic units for voice conversion
Andy T Liu, Po-chun Hsu, and Hung-yi Lee · 2019
Closest in time.
Language learning using speech to image retrieval
Danny Merkx, Stefan L. Frank, and Mirjam Ernestus · 2019
Closest in time.
On the contributions of visual and textual supervision in low-resource semantic speech retrieval
Ankita Pasad, Bowen Shi, Herman Kamper, and Karen Livescu · 2019
Closest in time.
Learning problem-agnostic speech representations from multiple self-supervised tasks
Santiago Pascual, Mirco Ravanelli, Joan Serrà, Antonio Bonafonte, and Yoshua Bengio · 2019
Closest in time.
Generating diverse high-fidelity images with vq-vae-2
Original
Ali Razavi, Aaron van den Oord, and Oriol Vinyals · 2019
Closest in time.
Learning words by drawing images
Dídac Surís, Adrià Recasens, David Bau, David Harwath, James Glass, and Antonio Torralba · 2019
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…