2019

Gmail Smart Compose: Real-Time Assisted Writing

Chen, Mia Xu, Lee, Benjamin N, Bansal, Gagan et al.

Understand

In this paper, we present Smart Compose, a novel system for generating interactive, real-time suggestions in Gmail that assists users in writing mails by reducing repetitive typing.

In the design and deployment of such a large-scale and complicated system, we faced several challenges including model selection, performance evaluation, serving and other practical issues.
At the core of Smart Compose is a large-scale neural language model.
We leveraged state-of-the-art machine learning techniques for language model training which enabled high-quality suggestion prediction, and constructed novel serving infrastructure for high-throughput and real-time inference.

Built on

Cross-lingual Language Model Pretraining
Original
Guillaume Lample and Alexis Conneu. 2019 · 1901
Earlier work this paper cites.
Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modeling
Jonathan Shen, Patrick Nguyen, Yonghui Wu, Zhifeng Chen, et al · 1902
Earlier work this paper cites.
Estimation of probabilities from sparse data for the language model component of a speech recognizer
Slava Katz. 1987 · 1987
Earlier work this paper cites.
Usability Engineering
Jakob Nielsen. 1993 · 1993
Earlier work this paper cites.
Long short-term memory
Sepp Hochreiter and Jürgen Schmidhuber. 1997 · 1997
Earlier work this paper cites.
Statistical Methods for Speech Recognition
Fred Jelinek. 1997 · 1997
Earlier work this paper cites.
Foundations of Statistical Natural Language Processing
Christopher Manning and Hinrich Schütze. 1999 · 1999
Earlier work this paper cites.
Speech and Language Processing
Daniel Jurafsky and James H. Martin. 2000 · 2000
Earlier work this paper cites.
Language Modeling for Soft Keyboards. In Proceedings of the 7th International Conference on Intelligent User Interfaces . ACM, New York, NY, USA
Joshua Goodman, Gina Venolia, Keith Steury, and Chauncey Parker. 2002 · 2002
Earlier work this paper cites.
Generalized Algorithms for Constructing Statistical Language Models. In Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics
Cyril Allauzen, Mehryar Mohri, and Brian Roark. 2003 · 2003
Earlier work this paper cites.
A Neural Probabilistic Language Model
Yoshua Bengio, Réjean Ducharme, Pascal Vincent, , and Christian Jauvin. 2003 · 2003
Earlier work this paper cites.
Large language models in machine translation. In Proceedings of Empirical Methods in Natural Language Processing
Thorsten Brants, Ashok C. Popat, Peng Xu, Franz J. Och, and Jeffrey Dean. 2007 · 2007
Earlier work this paper cites.
Large-Scale Distributed Language Modeling. In 2007 IEEE International Conference on Acoustics, Speech and Signal Processing - ICASSP ’07 , Vol. 4
Ahmad Emami, Kishore Papineni, and Jeffrey Sorensen. 2007 · 2007
Earlier work this paper cites.
Generalized linear interpolation of language models. In 2007 IEEE Workshop on Automatic Speech Recognition Understanding (ASRU) . 136–140
Bo-June Hsu. 2007 · 2007
Earlier work this paper cites.
Recurrent neural network based language model. In INTERSPEECH 2010, 11th Annual Conference of the International Speech Communication Association, Makuhari, Chiba, Japan, September 26-30, 2010
Tomas Mikolov, Martin Karafiát, Lukás Burget, Jan Cernocký, and Sanjeev Khudanpur. 2010 · 2010
Earlier work this paper cites.
Statistical Machine Translation
Philipp Koehn. 2011 · 2011
Earlier work this paper cites.
Japanese and Korean voice search. In 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . 5149–5152
M. Schuster and K. Nakajima. 2012 · 2012
Earlier work this paper cites.

Similar

Auto-Encoding Variational Bayes
Original
Diederik Kingma and Max Welling. 2014 · 2014
Cited alongside, same era.
Adam: A Method for Stochastic Optimization
Original
Diederik P. Kingma and Jimmy Ba. 2014 · 2014
Cited alongside, same era.
Sequence to Sequence Learning with Neural Networks
Ilya Sutskever, Oriol Vinyals, and Quoc Le. 2014 · 2014
Cited alongside, same era.
Investigation of back-off based interpolation between recurrent neural network and n-gram language models. In 2015 IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) . 181–186
X. Chen, X. Liu, M. J. F. Gales, and P. C. Woodland. 2015 · 2015
Cited alongside, same era.
Semi-supervised sequence learning. In Advances in Neural Information Processing Systems . 3079–3087
Andrew M. Dai and Quoc V. Le. 2015 · 2015
Cited alongside, same era.
Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015 · 2015
Cited alongside, same era.
Rethinking the Inception Architecture for Computer Vision
Original
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna. 2015 · 2015
Cited alongside, same era.
Personalizing universal recurrent neural network language model with user characteristic features by social network crowdsourcing. In 2015 IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) . 84–91
Bo-Hsiang Tseng, Hung-yi Lee, and Lin-Shan Lee. 2015 · 2015
Cited alongside, same era.
Generating sentences from a continuous space. In 20th SIGNLL Conference on Computational Natural Language Learning (CoNLL)
Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew M Dai, Rafal Jozefowicz, and Samy Bengio. 2016 · 2016
Cited alongside, same era.
Semantics derived automatically from language corpora necessarily contain human biases
Aylin Caliskan-Islam, Joanna Bryson, and Arvind Narayanan. 2016 · 2016
Cited alongside, same era.
Revisiting Distributed Synchronous SGD
Original
Jianmin Chen, Rajat Monga, Samy Bengio, and Rafal Józefowicz. 2016 · 2016
Cited alongside, same era.
Multi-Way, Multilingual Neural Machine Translation with a Shared Attention Mechanism. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies . Association for Computational Linguistics, 866–875
Orhan Firat, Kyunghyun Cho, and Yoshua Bengio. 2016 · 2016
Cited alongside, same era.

Then

Google’s Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation
Melvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2017 · 2017
Later among the works it cites.
In-Datacenter Performance Analysis of a Tensor Processing Unit
Original
Norman P. Jouppi, Cliff Young, Nishant Patil, and et al David A. Patterson. 2017 · 2017
Later among the works it cites.
On the State of the Art of Evaluation in Neural Language Models
Original
Gábor Melis, Chris Dyer, and Phil Blunsom. 2017 · 2017
Later among the works it cites.
Attention is All You Need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017 · 2017
Later among the works it cites.
Efficient Transfer Learning Schemes for Personalized Language Modeling using Recurrent Neural Network
Original
Seunghyun Yoon, Hyeongu Yun, Yuna Kim, Gyu-tae Park, and Kyomin Jung. 2017 · 2017
Later among the works it cites.
Lightweight Adaptive Mixture of Neural and N-gram Language Models
Original
Anton Bakhtin, Arthur Szlam, Marc’Aurelio Ranzato, and Edouard Grave. 2018 · 2018
Later among the works it cites.
The Secret Sharer: Measuring Unintended Neural Network Memorization & Extracting Secrets
Original
Nicholas Carlini, Chang Liu, Jernej Kos, Úlfar Erlingsson, and Dawn Song. 2018 · 2018
Later among the works it cites.
The Best of Both Worlds: Combining Recent Advances in Neural Machine Translation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) . Association for Computational Linguistics, 76–86
Mia Xu Chen, Orhan Firat, Ankur Bapna, Melvin Johnson, Wolfgang Macherey, George Foster, Llion Jones, Mike Schuster, Noam Shazeer, Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Zhifeng Chen, Yonghui Wu, and Macduff Hughes. 2018 · 2018
Later among the works it cites.
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Original
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018 · 2018
Later among the works it cites.
Personalized Language Model for Query Auto-Completion. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) . Association for Computational Linguistics
Aaron Jaech and Mari Ostendorf. 2018 · 2018
Later among the works it cites.
Deep contextualized word representations. In Proc. of NAACL
Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018 · 2018
Later among the works it cites.
Breaking the Softmax Bottleneck: A High-Rank RNN Language Model. In International Conference on Learning Representations
Zhilin Yang, Zihang Dai, Ruslan Salakhutdinov, and William W. Cohen. 2018 · 2018
Later among the works it cites.
Fraternal Dropout. In International Conference on Learning Representations
Konrad Zolna, Devansh Arpit, Dendi Suhubdy, and Yoshua Bengio. 2018 · 2018
Later among the works it cites.
Google Cloud TPU
Google. 2019a · 2019
Closest in time.
XLA Compiler
Google. 2019b · 2019
Closest in time.
Email Statistics Report, 2018-2022
S. Radicati. 2018 · 2019
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…