Fetching the paper…

Reading the bibliography…

2021

Scaling Laws for Acoustic Models

Droppo, Jasha, Elibol, Oguz

Original
PDF
Post on X

Understand

There is a recent trend in machine learning to increase model quality by growing models to sizes previously thought to be unreasonable.

Recent work has shown that autoregressive generative models with cross-entropy objective functions exhibit smooth power-law relationships, or scaling laws, that predict model quality from model size, training set size, and the available compute budget.
These scaling laws allow one to choose nearly optimal hyper-parameters given constraints on available training data, model parameter count, or training computation budget.
In this paper, we demonstrate that acoustic models trained with an auto-predictive coding loss behave as if they are subject to similar scaling laws.

Scaling Laws for Acoustic Models · Around

Built on

A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever, “Improving language understanding with unsupervised learning,” OpenAI, Tech. Rep., 2018
2018
Earlier work this paper cites.
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” arXiv e-prints , p. arXiv:1810.04805, Oct. 2018
Original
2018
Earlier work this paper cites.
A. van den Oord, Y. Li, and O. Vinyals, “Representation Learning with Contrastive Predictive Coding,” arXiv e-prints , p. arXiv:1807.03748, Jul. 2018
Original
2018
Earlier work this paper cites.
J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei, “Scaling Laws for Neural Language Models,” arXiv e-prints , p. arXiv:2001.08361, Jan. 2020
Original
2020
Earlier work this paper cites.
T. Henighan, J. Kaplan, M. Katz, M. Chen, C. Hesse, J. Jackson, H. Jun, T. B. Brown, P. Dhariwal, S. Gray, C. Hallacy, B. Mann, A. Radford, A. Ramesh, N. Ryder, D. M. Ziegler, J. Schulman, D. Amodei, and S. McCandlish, “Scaling Laws for Autoregressive Generative Modeling,” arXiv e-prints , p. arXiv:2010.14701, Oct. 2020
Original
2020
Earlier work this paper cites.

Similar

T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei, “Language models are few-shot learners,” 2020
2020
Cited alongside, same era.
S. Ling and Y. Liu, “DeCoAR 2.0: Deep Contextualized Acoustic Representations with Vector Quantization,” arXiv e-prints , p. arXiv:2012.06659, Dec. 2020
Original
2020
Cited alongside, same era.
Y. Chung and J. Glass, “Generative pre-training for speech with autoregressive predictive coding,” in ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , 2020, pp. 3497–3501
2020
Cited alongside, same era.
Y.-A. Chung, H. Tang, and J. Glass, “Vector-quantized autoregressive predictive coding,” October 2020, pp. 3760–3764

Then

A. Baevski, H. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” 2020
2020
Later among the works it cites.
Q. Xu, A. Baevski, T. Likhomanenko, P. Tomasello, A. Conneau, R. Collobert, G. Synnaeve, and M. Auli, “Self-training and Pre-training are Complementary for Speech Recognition,” arXiv e-prints , p. arXiv:2010.11430, Oct. 2020
Original
2020
Later among the works it cites.
S. Ling, Y. Liu, J. Salazar, and K. Kirchhoff, “Deep contextualized acoustic representations for semi-supervised speech recognition,” in ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , 2020, pp. 6429–6433
2020
Later among the works it cites.
W. Fedus, B. Zoph, and N. Shazeer, “Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity,” arXiv e-prints , p. arXiv:2101.03961, Jan. 2021
Original
2021
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

2020

Cited alongside, same era.