Fetching the paper…
Reading the bibliography…
We use neural ordinary differential equations to formulate a variant of the Transformer that is depth-adaptive in the sense that an input-dependent number of time steps is taken by the ordinary differential equation solver.
Dropout: a simple way to prevent neural networks from overfitting
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov · 2014
Earlier work this paper cites.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton · 2016
Earlier work this paper cites.
Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin · 2017
Earlier work this paper cites.
Neural ordinary differential equations
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud · 2018
Earlier work this paper cites.
Generating long sequences with sparse transformers
Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever · 2019
Earlier work this paper cites.
Universal transformers
Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser · 2019
Cited alongside, same era.
A survey on document-level machine translation: Methods and evaluation
Sameen Maruf, Fahimeh Saleh, and Gholamreza Haffari · 2019
Cited alongside, same era.
Scalable reversible generative models with free-form continuous dynamics
Will Grathwohl, Ricky T. Q. Chen, Jesse Bettencourt, and David Duvenaud · 2019
Cited alongside, same era.
Reformer: The efficient transformer
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya · 2020
Cited alongside, same era.
Longformer: The long-document transformer
Iz Beltagy, Matthew E. Peters, and Arman Cohan · 2020
Cited alongside, same era.
Depth-adaptive transformer
Maha Elbayad, Jiatao Gu, Edouard Grave, and Michael Auli · 2020
Closest in time.
Theoretical limitations of self-attention in neural sequence models
Michael Hahn · 2020
Closest in time.
Ot-flow: Fast and accurate continuous normalizing flows via optimal transport
Derek Onken, Samy Wu Fung, Xingjian Li, and Lars Ruthotto · 2020
Closest in time.
Learning differential equations that are easy to solve
Jacob Kelly, Jesse Bettencourt, Matthew James Johnson, and David Duvenaud · 2020
Closest in time.
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…