Fetching the paper…
Reading the bibliography…
We investigate forward signal propagation and gradient back propagation in deep, randomly initialized transformers, yielding simple necessary and sufficient conditions on initialization hyperparameters that ensure trainability of deep transformers.
Food-101 – mining discriminative components with random forests
Bossard, L., Guillaumin, M., and Van Gool, L · 2014
Earlier work this paper cites.
Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
Saxe, A., McClelland, J., and Ganguli, S · 2014
Earlier work this paper cites.
Exponential expressivity in deep neural networks through transient chaos
Poole, B., Lahiri, S., Raghu, M., Sohl-Dickstein, J., and Ganguli, S · 2016
Earlier work this paper cites.
Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice
Pennington, J., Schoenholz, S., and Ganguli, S · 2017
Earlier work this paper cites.
Deep information propagation
Schoenholz, S. S., Gilmer, J., Ganguli, S., and Sohl-Dickstein, J · 2017
Earlier work this paper cites.
Attention is all you need
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I · 2017
Earlier work this paper cites.
Bert: Pre-training of deep bidirectional transformers for language understanding
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K · 2018
Earlier work this paper cites.
The emergence of spectral universality in deep networks
Pennington, J., Schoenholz, S. S., and Ganguli, S · 2018
Cited alongside, same era.
Language models are few-shot learners
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al · 2020
Cited alongside, same era.
An image is worth 16x16 words: Transformers for image recognition at scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al · 2020
Cited alongside, same era.
Vivit: A video vision transformer
Arnab, A., Dehghani, M., Heigold, G., Sun, C., Lučić, M., and Schmid, C · 2021
Cited alongside, same era.
Attention is not all you need: Pure attention loses rank doubly exponentially with depth
Dong, Y., Cordonnier, J.-B., and Loukas, A · 2021
Cited alongside, same era.
Autoinit: Automatic initialization via jacobian tuning
He, T., Doshi, D., and Gromov, A · 2022
Later among the works it cites.
Transformers in vision: A survey
Khan, S., Naseer, M., Hayat, M., Zamir, S. W., Khan, F. S., and Shah, M · 2022
Later among the works it cites.
Signal propagation in transformers: Theoretical perspectives and the role of rank collapse
Noci, L., Anagnostidis, S., Biggio, L., Orvieto, A., Singh, S. P., and Lucchi, A · 2022
Later among the works it cites.
Effective theory of transformers at initialization
Dinan, E., Yaida, S., and Zhang, S · 2023
Later among the works it cites.
Critical initialization of wide and deep neural networks using partial jacobians: General theory and applications
Doshi, D., He, T., and Gromov, A · 2023
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Martens, J., Ballard, A., Desjardins, G., Swirszcz, G., Dalibard, V., Sohl-Dickstein, J., and Schoenholz, S. S · 2021
Cited alongside, same era.
A survey on vision transformer
Han, K., Wang, Y., Chen, H., Chen, X., Guo, J., Liu, Z., Tang, Y., Xiao, A., Xu, C., Xu, Y., et al · 2022
Cited alongside, same era.
A mathematical perspective on transformers
Geshkovski, B., Letrouit, C., Polyanskiy, Y., and Rigollet, P
Cited in the paper.
The emergence of clusters in self-attention dynamics
Geshkovski, B., Letrouit, C., Polyanskiy, Y., and Rigollet, P
Cited in the paper.
Later among the works it cites.
Simplifying transformer blocks
He, B. and Hofmann, T · 2023
Later among the works it cites.
The shaped transformer: Attention models in the infinite depth-and-width limit
Noci, L., Li, C., Li, M. B., He, B., Hofmann, T., Maddison, C., and Roy, D. M · 2023
Later among the works it cites.