Fetching the paper…

Reading the bibliography…

2023

Effective Theory of Transformers at Initialization

Dinan, Emily, Yaida, Sho, Zhang, Susan

Original
PDF
Post on X

Understand

We perform an effective-theory analysis of forward-backward signal propagation in wide and deep Transformers, i.e., residual neural networks with multi-head self-attention blocks and multilayer perceptron blocks.

This analysis suggests particular width scalings of initialization and training hyperparameters for these models.
We then take up such suggestions, training Vision and Language Transformers in practical setups.

Effective Theory of Transformers at Initialization · Around

Built on

Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, “RoBERTa: A Robustly Optimized BERT Pretraining Approach,” arXiv:1907.11692 [cs.CL]
Original
1907
Earlier work this paper cites.
B. T. Polyak and A. B. Juditsky, “Acceleration of Stochastic Approximation by Averaging,” SIAM Journal on Control and Optimization
1992
Earlier work this paper cites.
Springer, 1996
R. M. Neal, “Priors for Infinite Networks,” in Bayesian Learning for Neural Networks · 1996
Earlier work this paper cites.
N. Shazeer, “GLU Variants Improve Transformer,” arXiv:2002.05202 [cs.CL]
Original
2002
Earlier work this paper cites.

Similar

O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, et al
2015
Cited alongside, same era.
Ravenio Books, 2016
G. K. Zipf, Human Behavior and the Principle of Least Effort: An Introduction to Human Ecology · 2016
Cited alongside, same era.
http://commoncrawl.org/2016/10/news-dataset-available
S. Nagel, “CC-News,” 2016 · 2016
Cited alongside, same era.
R. Wightman et al
2019
Cited alongside, same era.
J. W. Rae, S. Borgeaud, T. Cai, K. Millican, J. Hoffmann, F. Song, J. Aslanides, S. Henderson, R. Ring, S. Young, et al
Original
Cited in the paper.
S. Smith, M. Patwary, B. Norick, P. LeGresley, S. Rajbhandari, J. Casper, Z. Liu, S. Prabhumoye, G. Zerveas, V. Korthikanti, et al
Original
Cited in the paper.
A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, et al
Original
Cited in the paper.
S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, et al
Original
Cited in the paper.
J. L. Ba, J. R. Kiros, and G. E. Hinton, “Layer Normalization,” arXiv:1607.06450 [stat.ML]
Original
Cited in the paper.
D. Doshi, T. He, and A. Gromov, “Critical Initialization of Wide and Deep Neural Networks through Partial Jacobians: General Theory and Applications,” arXiv:2111.12143 [cs.LG]
Original
Cited in the paper.
S. Yaida, “Meta-Principled Family of Hyperparameter Scaling Strategies,” arXiv:2210.04909 [cs.LG]
Original
Cited in the paper.
G. Yang, E. J. Hu, I. Babuschkin, S. Sidor, X. Liu, D. Farhi, N. Ryder, J. Pachocki, W. Chen, and J. Gao, “Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer,” arXiv:2203.03466 [cs.LG]
Original
Cited in the paper.

Then

http://Skylion007.github.io/OpenWebTextCorpus
A. Gokaslan and V. Cohen, “Openwebtext corpus,” 2019 · 2019
Later among the works it cites.
A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al
2019
Later among the works it cites.
Cambridge University Press, 2022
Original
D. A. Roberts, S. Yaida, and B. Hanin, The Principles of Deep Learning Theory · 2022
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…