2017

Synthetic and Natural Noise Both Break Neural Machine Translation

Belinkov, Yonatan, Bisk, Yonatan

Understand

Character-based neural machine translation (NMT) models alleviate out-of-vocabulary issues, learn morphology, and move us closer to completely end-to-end translation systems.

Unfortunately, they are also very brittle and easily falter when presented with noisy data.
In this paper, we confront NMT models with synthetic and natural sources of noise.
We find that state-of-the-art models fail to translate even moderately noisy texts that humans have no trouble comprehending.

Built on

Perceptual recognition as a function of meaningfulness of stimulus material
G. M. Reicher · 1969
Earlier work this paper cites.
The significance of letter position in word recognition
G. E. Rawlinson · 1976
Earlier work this paper cites.
Word recognition inside out and outside in
L. X. McCusker, P. B. Gough, and R. G. Bias · 1981
Earlier work this paper cites.
Noise injection into inputs in back-propagation learning
K. Matsuoka · 1992
Earlier work this paper cites.
Training with noise is equivalent to Tikhonov regularization
Christopher Bishop · 1995
Earlier work this paper cites.
Long short-term memory
Sepp Hochreiter and Jürgen Schmidhuber · 1997
Earlier work this paper cites.
Disruption to word or letter processing? The origins of case-mixing effects
K. Mayall, G.W. Humphreys, and A. Olson · 1997
Earlier work this paper cites.
Cognitive restoration of reversed speech
Kourosh Saberi and David R. Perrott · 1999
Earlier work this paper cites.
The remarkable inefficiency of word recognition
D. G. Pelli, B. Farell, and D.C. Moore · 2003
Earlier work this paper cites.
Nightmare at Test Time: Robust Learning by Feature Deletion
Amir Globerson and Sam Roweis · 2006
Earlier work this paper cites.
Raeding Wrods With Jubmled Lettres: There Is a Cost
Keith Rayner, Sarah J. White, Rebecca L. Johnson, and Simon P. Liversedge · 2006
Earlier work this paper cites.
Casting out Demons: Sanitizing Training Data for Anomaly Sensors
Gabriela F. Cretu, Angelos Stavrou, Michael E. Locasto, Salvatore J. Stolfo, and Angelos D. Keromytis · 2008
Earlier work this paper cites.
ANTIDOTE: Understanding and Defending Against Poisoning of Anomaly Detectors
Benjamin I.P. Rubinstein, Blaine Nelson, Ling Huang, Anthony D. Joseph, Shing-hon Lau, Satish Rao, Nina Taft, and J. D. Tygar · 2009
Earlier work this paper cites.
Mining Naturally-occurring Corrections and Paraphrases from Wikipedia’s Revision History
Aurélien Max and Guillaume Wisniewski · 2010
Earlier work this paper cites.
Poisoning Attacks Against Support Vector Machines
Battista Biggio, Blaine Nelson, and Pavel Laskov · 2012
Earlier work this paper cites.
WIT 3 : Web Inventory of Transcribed and Translated Talks
Mauro Cettolo, Christian Girardi, and Marcello Federico · 2012
Earlier work this paper cites.
Measuring Contextual Fitness Using Error Contexts Extracted from the Wikipedia Revision History
Torsten Zesch · 2012
Earlier work this paper cites.

Similar

MERLIN: An online trilingual learner corpus empirically grounding the European Reference Levels in authentic learner data, 10 2013
Katrin Wisniewski, Karin Schöne, Lionel Nicolas, Chiara Vettori, Adriane Boyd, Detmar Meurers, Andrea Abel, and Jirka Hana · 2013
Cited alongside, same era.
Neural Machine Translation by Jointly Learning to Align and Translate
Original
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio · 2014
Cited alongside, same era.
Sequence to Sequence Learning with Neural Networks
Ilya Sutskever, Oriol Vinyals, and Quoc VV Le · 2014
Cited alongside, same era.
Intriguing properties of neural networks
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus · 2014
Cited alongside, same era.
Explaining and Harnessing Adversarial Examples
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy · 2015
Cited alongside, same era.
Character-aware Neural Language Models
Original
Yoon Kim, Yacine Jernite, David Sontag, and Alexander M Rush · 2015
Cited alongside, same era.
Using Machine Teaching to Identify Optimal Training-set Attacks on Machine Learners
Shike Mei and Xiaojin Zhu · 2015
Cited alongside, same era.
Large-Scale Machine Translation between Arabic and Hebrew: Available Corpora and Initial Results
Yonatan Belinkov and James Glass · 2016
Cited alongside, same era.
The University of Edinburgh’s systems submission to the MT task at IWSLT
Marcin Junczys-Dowmunt and Alexandra Birch · 2016
Cited alongside, same era.
Seq2seq-attn
Yoon Kim · 2016
Cited alongside, same era.
Crafting Adversarial Input Sequences for Recurrent Neural Networks
Nicolas Papernot, Patrick McDaniel, Ananthram Swami, and Richard Harang · 2016
Cited alongside, same era.
Does String-Based Neural MT Learn Source Syntax?
Xing Shi, Inkit Padhi, and Kevin Knight · 2016
Cited alongside, same era.

Then

How Robust Are Character-Based Word Embeddings in Tagging and MT Against Wrod Scramlbing or Randdm Nouse?
Original
Georg Heigold, Günter Neumann, and Josef van Genabith · 2017
Closest in time.
Adversarial Examples for Evaluating Reading Comprehension Systems
Robin Jia and Percy Liang · 2017
Closest in time.
Fully Character-Level Neural Machine Translation without Explicit Segmentation
Jason Lee, Kyunghyun Cho, and Thomas Hofmann · 2017
Closest in time.
Deep Text Classification Can be Fooled
Original
Bin Liang, Hongcheng Li, Miaoqiang Su, Pan Bian, Xirong Li, and Wenchang Shi · 2017
Closest in time.
Delving into Transferable Adversarial Examples and Black-box Attacks
Yanpei Liu, Xinyun Chen, Chang Liu, and Dawn Song · 2017
Closest in time.
Simple Black-Box Adversarial Attacks on Deep Neural Networks
N. Narodytska and S. Kasiviswanathan · 2017
Closest in time.
Practical Black-Box Attacks Against Machine Learning
Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z. Berkay Celik, and Ananthram Swami · 2017
Closest in time.
Challenging Language-Dependent Segmentation for Arabic: An Application to Machine Translation and Part-of-Speech Tagging
Hassan Sajjad, Fahim Dalvi, Nadir Durrani, Ahmed Abdelali, Yonatan Belinkov, and Stephan Vogel · 2017
Closest in time.
Robsut Wrod Reocginiton via Semi-Character Recurrent Neural Network
Keisuke Sakaguchi, Kevin Duh, Matt Post, and Benjamin Van Durme · 2017
Closest in time.
Towards Crafting Text Adversarial Samples
Original
Suranjana Samanta and Sameep Mehta · 2017
Closest in time.
CzeSL grammatical error correction dataset (CzeSL-GEC)
Karel Šebesta, Zuzanna Bedřichová, Kateřina Šormová, Barbora Štindlová, Milan Hrdlička, Tereza Hrdličková, Jiří Hana, Vladimír Petkevič, Tomáš Jelínek, Svatava Škodová, Petr Janeš, Kateřina Lundáková, Hana Skoumalová, Šimon Sládek, Piotr Pierscieniak, Dagmar Toufarová, Milan Straka, Alexandr Rosen, Jakub Náplava, and Marie Poláčková · 2017
Closest in time.
Nematus: a Toolkit for Neural Machine Translation
Rico Sennrich, Orhan Firat, Kyunghyun Cho, Alexandra Birch, Barry Haddow, Julian Hitschler, Marcin Junczys-Dowmunt, Samuel Läubli, Antonio Valerio Miceli Barone, Jozef Mokry, and Maria Nadejde · 2017
Closest in time.
Ensemble Adversarial Training: Attacks and Defenses
Original
Florian Tramèr, Alexey Kurakin, Nicolas Papernot, Dan Boneh, and Patrick McDaniel · 2017
Closest in time.
Black-box Generation of Adversarial Text Sequences to Evade Deep Learning Classifiers
Original
Ji Gao, Jack Lanchantin, Mary Lou Soffa, and Yanjun Qi · 2018
Closest in time.
Character-based Neural Machine Translation
Marta R. Costa-jussà and José A. R. Fonollosa · 2058
Closest in time.
How Grammatical is Character-level Neural Machine Translation? Assessing MT Quality with Contrastive Translation Pairs
Rico Sennrich · 2060
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…