2018

Has Machine Translation Achieved Human Parity? A Case for Document-level Evaluation

Läubli, Samuel, Sennrich, Rico, Volk, Martin

Understand

Recent research suggests that neural machine translation achieves parity with professional human translation on the WMT Chinese--English news translation task.

We empirically test this claim with alternative evaluation protocols, contrasting the evaluation of single sentences and entire documents.
In a pairwise ranking experiment, human raters assessing adequacy and fluency show a stronger preference for human over machine translation when evaluating documents as compared to isolated sentences.
Our findings emphasise the need to shift towards document-level evaluation as machine translation improves to the degree that errors which are hard or impossible to spot at the sentence-level become decisive in discriminating quality of different translation outputs.

Built on

Another Look at the Sign Test When Ties Are Present: The Problem of Confidence Intervals
John D. Emerson and Gary A. Simon. 1979 · 1979
Earlier work this paper cites.
Critical study of methods for evaluating the quality of machine translation
Georges Van Slype. 1979 · 1979
Earlier work this paper cites.
Was in den Köpfen von Übersetzern vorgeht
Hans P. Krings. 1986 · 1986
Earlier work this paper cites.
(Meta-) evaluation of machine translation
Chris Callison-Burch, Cameron Fordyce, Philipp Koehn, Christof Monz, and Josh Schroeder. 2007 · 2007
Earlier work this paper cites.
Crowdsourcing User Studies with Mechanical Turk
Aniket Kittur, Ed H. Chi, and Bongwon Suh. 2008 · 2008
Earlier work this paper cites.
Fast, cheap, and creative: Evaluating translation quality using amazon’s mechanical turk
Chris Callison-Burch. 2009 · 2009
Earlier work this paper cites.
Continuous Measurement Scales in Human Evaluation of Machine Translation
Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2013 · 2013
Earlier work this paper cites.
Recurrent Continuous Translation Models
Nal Kalchbrenner and Phil Blunsom. 2013 · 2013
Earlier work this paper cites.

Similar

Sequence to Sequence Learning with Neural Networks
Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014 · 2014
Cited alongside, same era.
Neural Machine Translation by Jointly Learning to Align and Translate
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015 · 2015
Cited alongside, same era.
Stanford Neural Machine Translation Systems for Spoken Language Domains
Minh-Thang Luong and Christopher D. Manning. 2015 · 2015
Cited alongside, same era.
Ten years of WMT evaluation campaigns: Lessons learnt
Ondrej Bojar, Christian Federmann, Barry Haddow, Philipp Koehn, Matt Post, and Lucia Specia. 2016a · 2016
Cited alongside, same era.
Findings of the 2016 Conference on Machine Translation (WMT16)
Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Aurelie Neveol, Mariana Neves, Martin Popel, Matt Post, Raphael Rubino, Carolina Scarton, Lucia Specia, Marco Turchi, Karin Verspoor, and Marcos Zampieri. 2016b · 2016
Cited alongside, same era.
Kyoto University participation to WAT 2016
Fabien Cromieres, Chenhui Chu, Toshiaki Nakazawa, and Sadao Kurohashi. 2016 · 2016
Cited alongside, same era.
Edinburgh Neural Machine Translation Systems for WMT 16
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016 · 2016
Cited alongside, same era.
Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Original
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2016 · 2016
Cited alongside, same era.
Is Neural Machine Translation the New State of the Art?
Sheila Castilho, Joss Moorkens, Federico Gaspari, Iacer Calixto, John Tinsley, and Andy Way. 2017a
Cited in the paper.
A Comparative Quality Evaluation of PBSMT and NMT using Professional Translators
Sheila Castilho, Joss Moorkens, Federico Gaspari, Rico Sennrich, Vilelmini Sosoni, Yota Georgakopoulou, Pintu Lohar, Andy Way, Antonio Valerio Miceli Barone, and Maria Gialama. 2017b
Cited in the paper.

Then

Findings of the 2017 Conference on Machine Translation (WMT17)
Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Shujian Huang, Matthias Huck, Philipp Koehn, Qun Liu, Varvara Logacheva, Christof Monz, Matteo Negri, Matt Post, Raphael Rubino, Lucia Specia, and Marco Turchi. 2017 · 2017
Later among the works it cites.
Overview of the IWSLT 2017 Evaluation Campaign
Mauro Cettolo, Marcello Federico, Luisa Bentivogli, Jan Niehues, Sebastian Stüker, Katsuitho Sudoh, Koichiro Yoshino, and Christian Federmann. 2017 · 2017
Later among the works it cites.
Can machine translation systems be evaluated by the crowd alone?
Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2017 · 2017
Later among the works it cites.
The University of Edinburgh’s Neural MT Systems for WMT17
Rico Sennrich, Alexandra Birch, Anna Currey, Ulrich Germann, Barry Haddow, Kenneth Heafield, Antonio Valerio Miceli Barone, and Philip Williams. 2017 · 2017
Later among the works it cites.
Attention is All you Need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017 · 2017
Later among the works it cites.
Proceedings of the Third Workshop on Discourse in Machine Translation . Copenhagen, Denmark
Bonnie Webber, Andrei Popescu-Belis, and Jörg Tiedemann, editors. 2017 · 2017
Later among the works it cites.
Achieving Human Parity on Automatic Chinese to English News Translation
Original
Hany Hassan, Anthony Aue, Chang Chen, Vishal Chowdhary, Jonathan Clark, Christian Federmann, Xuedong Huang, Marcin Junczys-Dowmunt, William Lewis, Mu Li, Shujie Liu, Tie-Yan Liu, Renqian Luo, Arul Menezes, Tao Qin, Frank Seide, Xu Tan, Fei Tian, Lijun Wu, Shuangzhi Wu, Yingce Xia, Dongdong Zhang, Zhirui Zhang, and Ming Zhou. 2018 · 2018
Closest in time.
Attaining the Unattainable? Reassessing Claims of Human Parity in Neural Machine Translation
Antonio Toral, Sheila Castilho, Ke Hu, and Andy Way. 2018 · 2018
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…