2019

The Evolved Transformer

So, David R., Liang, Chen, Le, Quoc V.

Understand

Recent works have highlighted the strength of the Transformer architecture on sequence tasks while, at the same time, neural architecture search (NAS) has begun to outperform human-designed models.

Our goal is to apply NAS to search for a better alternative to the Transformer.
We first construct a large search space inspired by the recent advances in feed-forward sequence models and then run evolutionary architecture search with warm starting by seeding our initial population with the Transformer.
To directly search on the computationally expensive WMT 2014 English-German translation task, we develop the Progressive Dynamic Hurdles method, which allows us to dynamically allocate more resources to more promising candidate models.

Built on

A comparative analysis of selection schemes used in genetic algorithms
Goldberg, D. E. and Deb, K · 1991
Earlier work this paper cites.
Long short-term memory
Hochreiter, S. and Schmidhuber, J · 1997
Earlier work this paper cites.
Imagenet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L · 2009
Earlier work this paper cites.
Learning multiple layers of features from tiny images
Krizhevsky, A. and Hinton, G · 2009
Earlier work this paper cites.
One billion word benchmark for measuring progress in statistical language modeling
Original
Chelba, C., Mikolov, T., Schuster, M., Ge, Q., Brants, T., and Koehn, P · 2013
Earlier work this paper cites.
Rectifier nonlinearities improve neural network acoustic models
Maas, A. L., Hannun, A. Y., and Ng, A. Y · 2013
Earlier work this paper cites.
On the properties of neural machine translation: Encoder–decoder approaches
Cho, K., Merrienboer, B. V., Bahdanau, D., and Bengio, Y · 2014
Earlier work this paper cites.
Sequence to sequence learning with neural networks
Sutskever, I., Vinyals, O., and Le, Q. V · 2014
Earlier work this paper cites.
Neural machine translation by jointly learning to align and translate
Bahdanau, D., Cho, K., and Bengio, Y · 2015
Earlier work this paper cites.
Semi-supervised sequence learning
Dai, A. M. and Le, Q. V · 2015
Earlier work this paper cites.
Layer normalization
Original
Ba, L. J., Kiros, R., and Hinton, G. E · 2016
Earlier work this paper cites.
Designing neural network architectures using reinforcement learning
Baker, B., Gupta, O., Naik, N., and Raskar, R · 2016
Earlier work this paper cites.
Non-stochastic best arm identification and hyperparameter optimization
Jamieson, K. G. and Talwalkar, A. S · 2016
Earlier work this paper cites.
Wavenet: A generative model for raw audio
Original
Van Den Oord, A., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., and Kavukcuoglu, K · 2016
Earlier work this paper cites.
Google’s neural machine translation system: Bridging the gap between human and machine translation
Original
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., Klingner, J., Shah, A., Johnson, M., Liu, X., Kaiser, L., Gouws, S., Kato, Y., Kudo, T., Kazawa, H., Stevens, K., Kurian, G., Patil, N., Wang, W., Young, C., Smith, J., Riesa, J., Rudnick, A., Vinyals, O., Corrado, G. S., Hughes, M., and Dean, J · 2016
Earlier work this paper cites.

Similar

Weighted transformer network for machine translation
Original
Ahmed, K., Keskar, N. S., and Socher, R · 2017
Cited alongside, same era.
Language modeling with gated convolutional networks
Dauphin, Y. N., Fan, A., Auli, M., and Grangier, D · 2017
Cited alongside, same era.
Convolutional sequence to sequence learning
Gehring, J., Auli, M., Grangier, D., Yarats, D., and Dauphin, Y. N · 2017
Cited alongside, same era.
Hyperband: a novel bandit-based approach to hyperparameter optimization
Li, L., Jamieson, K. G., DeSalvo, G., Rostamizadeh, A., and Talwalkar, A. S · 2017
Cited alongside, same era.
Sgdr: Stochastic gradient descent with warm restarts
Loshchilov, I. and Hutter, F · 2017
Cited alongside, same era.
Searching for activation functions
Original
Ramachandran, P., Zoph, B., and Le, Q. V · 2017
Cited alongside, same era.
Large-scale evolution of image classifiers
Real, E., Moore, S., Selle, A., Saxena, S., Suematsu, Y. L., Tan, J., Le, Q., and Kurakin, A · 2017
Cited alongside, same era.
Attention is all you need
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I · 2017
Cited alongside, same era.
Genetic cnn
Xie, L. and Yuille, A. L · 2017
Cited alongside, same era.
Neural architecture search with reinforcement learning
Zoph, B. and Le, Q. V · 2017
Cited alongside, same era.
SMASH: One-shot model architecture search through hypernetworks
Brock, A., Lim, T., Ritchie, J., and Weston, N · 2018
Cited alongside, same era.
The best of both worlds: Combining recent advances in neural machine translation
Original
Chen, M. X., Firat, O., Bapna, A., Johnson, M., Macherey, W., Foster, G., Jones, L., Parmar, N., Schuster, M., Chen, Z., Wu, Y., and Hughes, M · 2018
Cited alongside, same era.

Then

URL http://paracrawl.eu/download.html
ParaCrawl, 2018 · 2018
Later among the works it cites.
Deep contextualized word representations
Peters, M. E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., and Zettlemoyer, L · 2018
Later among the works it cites.
Efficient neural architecture search via parameter sharing
Pham, H., Guan, M. Y., Zoph, B., Le, Q. V., and Dean, J · 2018
Later among the works it cites.
A call for clarity in reporting BLEU scores
Original
Post, M · 2018
Later among the works it cites.
Improving language understanding by generative pre-training, 2018
Radford, A., Narasimhan, K., Salimans, T., and Sutskever, I · 2018
Later among the works it cites.
Self-attention with relative position representations
Original
Shaw, P., Uszkoreit, J., and Vaswani, A · 2018
Later among the works it cites.
Adafactor: adaptive learning rates with sublinear memory cost
Original
Shazeer, N. and Stern, M · 2018
Later among the works it cites.
Tensor2tensor for neural machine translation
Original
Vaswani, A., Bengio, S., Brevdo, E., Chollet, F., Gomez, A. N., Gouws, S., Jones, L., Łukasz Kaiser, Kalchbrenner, N., Parmar, N., Sepassi, R., Shazeer, N., and Uszkoreit, J · 2018
Later among the works it cites.
Fast and accurate reading comprehension by combining self-attention and convolution
Yu, A. W., Dohan, D., Luong, T., Zhao, R., Chen, K., Norouzi, M., and Le, Q. V · 2018
Later among the works it cites.
Towards automated deep learning: efficient joint neural architecture and hyperparameter search
Zela, A., Klein, A., Falkner, S., and Hutter, F · 2018
Later among the works it cites.
Learning transferable architectures for scalable image recognition
Zoph, B., Vasudevan, V., Shlens, J., and Le, Q. V · 2018
Later among the works it cites.
Regularized evolution for image classifier architecture search
Real, E., Aggarawal, A., Huang, Y., and Le, Q. V · 2019
Closest in time.
Pay less attention with lightweight and dynamic convolutions
Wu, F., Fan, A., Baevski, A., Dauphin, Y., and Auli, M · 2019
Closest in time.
SNAS: stochastic neural architecture search
Xie, S., Zheng, H., Liu, C., and Lin, L · 2019
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…