2017

Learning Joint Multilingual Sentence Representations with Neural Machine Translation

Schwenk, Holger, Douze, Matthijs

Understand

In this paper, we use the framework of neural machine translation to learn joint sentence representations across six very different languages.

Our aim is that a representation which is independent of the language, is likely to capture the underlying semantics.
We define a new cross-lingual similarity measure, compare up to 1.4M sentence representations and study the characteristics of close sentences.
We provide experimental evidence that sentences that are close in embedding space are indeed semantically highly related, but often have quite different structure and syntax.

Built on

A unified architecture for natural language processing: deep neural networks with multitask learning
Ronan Collobert and Jason Weston. 2008 · 2008
Earlier work this paper cites.
Multimodal deep learning
Jiquan Ngiam, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee, and Andrew Y. Ng. 2011 · 2011
Earlier work this paper cites.
Semi-supervised recursive autoencoders for predicting sentiment distributions
Richard Socher, Jeffrey Pennington, Eric H. Huang, Andrew Y. Ng, and Christopher D. Manning. 2011 · 2011
Earlier work this paper cites.
Wit 3 : Web inventory of transcribed and translated talks
Mauro Cettolo, Christian Girardi, and Marcello Federico. 2012 · 2012
Earlier work this paper cites.
Inducing crosslingual distributed representations of words
A. Klementiev, I. Titov, and B. Bhattarai. 2012 · 2012
Earlier work this paper cites.
Multilingual deep learning
Sarath Chandar, Mitesh M. Khapra, Balaraman Ravindran, Vikas Raykar, and Amrita Saha. 2013 · 2013
Earlier work this paper cites.
DeViSa:E a deep visual-semantic embedding model
Andrea Frome, Grep S. Corrado, Jonathon Shlens, Samy Bengio, Jeffrey Dean, marc’Aurelio Ranzato, and Thomas Mikolov. 2013 · 2013
Earlier work this paper cites.
Recurrent continuous translation models
Nal Kalchbrenner and Phil Blunsom. 2013 · 2013
Earlier work this paper cites.
Linguistic regularities in continuous word space representations
Tomas Mikolov, Wen tau Yih, and Geoffrey Zweig. 2013 · 2013
Earlier work this paper cites.
Learning phrase representations using rnn encoder-decoder for statistical machine translation
Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014 · 2014
Earlier work this paper cites.
Bilbowa: Fast bilingual distributed representations without word alignments
S. Gouews, Y. Bengio, and G. Corrado. 2014 · 2014
Earlier work this paper cites.
Multilingual models for compositional distributed semantics
Karl Moritz Hermann and Phil Blunsom. 2014 · 2014
Earlier work this paper cites.

Similar

Learning bilingual word representations by marginalizing alignments
T. Kociský, K.M. Hermann, and P. Blunsom. 2014 · 2014
Cited alongside, same era.
Sequence to sequence learning with neural networks
Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014 · 2014
Cited alongside, same era.
Neural machine translation by jointly learning to align and translate
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015 · 2015
Cited alongside, same era.
Trans-gram, fast cross-lingual word embeddings
J. Coulmance, J.M. Marty, G. Wenzek, and A. Benhaloum. 2015 · 2015
Cited alongside, same era.
Multi-task learning for multiple language translation
Daxiang Dong, Huan Wu, Wei He, Dianhai Yu, and Haifeng wang. 2015 · 2015
Cited alongside, same era.
Learning distributed representations for multilingual text sequences
Hieu Pham, Minh-Thang Luong, and Christopher D. Manning. 2015 · 2015
Cited alongside, same era.
Character-level convolutional networks for text classification
Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015 · 2015
Cited alongside, same era.
Self-adaptive hierarchical sentence model
Original
Han Zhao, Zhengdong Lu, and Pascal Poupart. 2015 · 2015
Cited alongside, same era.
Semeval-2016 task 1: Semantic textual similarity, monolingual and cross-lingual evaluation
Eneko Agirre, Carmen Banea, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Rada Mihalcea, German Rigau, and Janyce Wiebe. 2016 · 2016
Cited alongside, same era.
Correlation neural networks
Sarath Chandar, Mitesh M. Khapra, Hugo Larochelle, and Balaraman Ravindran. 2016 · 2016
Cited alongside, same era.
Google’s multilingual neural machine translation system: Enabling zero-shot translation
Original
Melvin Johnson et al. 2016 · 2016
Cited alongside, same era.
Learning natural language inference using bidirectional lstm model and inner-attention
Original
Yang Liu, Chenjie Sun, Lei Lin, and Xiaolong Wang. 2016 · 2016
Cited alongside, same era.

Then

Bilingual word embeddings fomr parallel and non-parallel corpora for cross-language classification
Aditua Mogadala and Achim Rettinger. 2016 · 2016
Later among the works it cites.
Zero-resource machine translation by multimodal encoder-decoder network with multimedia pivot
Original
Hideki Nakayama and Noriki Nishida. 2016 · 2016
Later among the works it cites.
A correlational encoder decoder architecture for pivot based sequence generation
Original
Amrita Saha, Mitesh M. Kharpa, Sarath Chandar, Janarthanan Rajendran, and Kyunghyun Cho. 2016 · 2016
Later among the works it cites.
Neural machine translation of rare words with subword units
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016 · 2016
Later among the works it cites.
Google’s neural machine translation system: Bridging the gap between human and machine translation
Original
Yonghui Wu et al. 2016 · 2016
Later among the works it cites.
The united nations parallel corpus v1.0
M Ziemski, Marcin Juncys-Dowmunt, and B. Pouliquen. 2016 · 2016
Later among the works it cites.
Multi-source neural translation
Barret Zoph and Kevin Knight. 2016 · 2016
Later among the works it cites.
A simple but tough-to-beat baseline for sentence embeddings
Sanjeev Arora, Yingyu Liang, and Tengyu Ma. 2017 · 2017
Closest in time.
Supervised learning of universal sentence representations from natural language inference data
Original
Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. 2017 · 2017
Closest in time.
Billion-scale similarity search with gpus
Original
Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2017 · 2017
Closest in time.
A structured self-attentive sentence embedding
Zhouhan Lin, Minwei Feng, Cicero Nogueira dos Santos, mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. 2017 · 2017
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…