2022

Data Scaling Laws in NMT: The Effect of Noise and Architecture

Bansal, Yamini, Ghorbani, Behrooz, Garg, Ankush et al.

Understand

In this work, we study the effect of varying the architecture and training data quality on the data scaling properties of Neural Machine Translation (NMT).

First, we establish that the test loss of encoder-decoder transformer models scales as a power law in the number of training samples, with a dependence on the model size.
Then, we systematically vary aspects of the training setup to understand how they impact the data scaling laws.
In particular, we change the following (1) Architecture and task setup: We compare to a transformer-LSTM hybrid, and a decoder-only transformer with a language modeling loss (2) Noise level in the training distribution: We experiment with filtering, and adding iid synthetic noise.

Built on

Four types of learning curves
Amari, S.-i., Fujita, N., and Shinomoto, S · 1992
Earlier work this paper cites.
Imagenet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., and Hinton, G. E · 2012
Earlier work this paper cites.
Improving neural machine translation models with monolingual data, 2016
Sennrich, R., Haddow, B., and Birch, A · 2016
Earlier work this paper cites.
Deep learning scaling is predictable, empirically
Original
Hestness, J., Narang, S., Ardalani, N., Diamos, G. F., Jun, H., Kianinejad, H., Patwary, M. M. A., Yang, Y., and Zhou, Y · 2017
Earlier work this paper cites.
The best of both worlds: Combining recent advances in neural machine translation, 2018
Chen, M. X., Firat, O., Bapna, A., Johnson, M., Macherey, W., Foster, G., Jones, L., Parmar, N., Schuster, M., Chen, Z., Wu, Y., and Hughes, M · 2018
Earlier work this paper cites.
Dual conditional cross-entropy filtering of noisy parallel corpora
Original
Junczys-Dowmunt, M · 2018
Earlier work this paper cites.
On the impact of various types of noise on neural machine translation
Khayrallah, H. and Koehn, P · 2018
Earlier work this paper cites.
Adafactor: Adaptive learning rates with sublinear memory cost, 2018
Shazeer, N. and Stern, M · 2018
Earlier work this paper cites.
Denoising neural machine translation training with trusted data and online data selection, 2018
Wang, W., Watanabe, T., Hughes, M., Nakagawa, T., and Chelba, C · 2018
Earlier work this paper cites.
Massively multilingual neural machine translation in the wild: Findings and challenges, 2019
Arivazhagan, N., Bapna, A., Firat, O., Lepikhin, D., Johnson, M., Krikun, M., Chen, M. X., Cao, Y., Foster, G., Cherry, C., Macherey, W., Chen, Z., and Wu, Y · 2019
Earlier work this paper cites.
Findings of the 2019 conference on machine translation (wmt19)
Barrault, L., Bojar, O., Costa-Jussa, M. R., Federmann, C., Fishel, M., Graham, Y., Haddow, B., Huck, M., Koehn, P., Malmasi, S., et al · 2019
Earlier work this paper cites.
APE at Scale and Its Implications on MT Evaluation Biases
Freitag, M., Caswell, I., and Roy, S · 2019
Earlier work this paper cites.
Neural machine translation enabling human parity innovations in the cloud
Microsoft · 2019
Earlier work this paper cites.
A constructive prediction of the generalization error across scales
Original
Rosenfeld, J. S., Rosenfeld, A., Belinkov, Y., and Shavit, N · 2019
Earlier work this paper cites.
The effect of translationese in machine translation test sets, 2019
Zhang, M. and Toral, A · 2019
Earlier work this paper cites.
ParaCrawl: Web-scale acquisition of parallel corpora
Bañón, M., Chen, P., Haddow, B., Heafield, K., Hoang, H., Esplà-Gomis, M., Forcada, M. L., Kamran, A., Kirefu, F., Koehn, P., Ortiz Rojas, S., Pla Sempere, L., Ramírez-Sánchez, G., Sarrías, E., Strelec, M., Thompson, B., Waites, W., Wiggins, D., and Zaragoza, J · 2020
Earlier work this paper cites.

Similar

Language models are few-shot learners
Original
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al · 2020
Cited alongside, same era.
Ccaligned: A massive collection of cross-lingual web-document pairs, 2020
El-Kishky, A., Chaudhary, V., Guzman, F., and Koehn, P · 2020
Cited alongside, same era.
Beyond english-centric multilingual machine translation, 2020
Fan, A., Bhosale, S., Schwenk, H., Ma, Z., El-Kishky, A., Goyal, S., Baines, M., Celebi, O., Wenzek, G., Chaudhary, V., Goyal, N., Birch, T., Liptchinsky, V., Edunov, S., Grave, E., Auli, M., and Joulin, A · 2020
Cited alongside, same era.
BLEU might be guilty but references are not innocent
Freitag, M., Grangier, D., and Caswell, I · 2020
Cited alongside, same era.
Recent advances in google translate
Google · 2020
Cited alongside, same era.
Statistical power and translationese in machine translation evaluation
Graham, Y., Haddow, B., and Koehn, P · 2020
Cited alongside, same era.
Scaling laws for neural language models
Original
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D · 2020
Cited alongside, same era.
Deep encoder, shallow decoder: Reevaluating non-autoregressive machine translation
Original
Kasai, J., Pappas, N., Peng, H., Cross, J., and Smith, N. A · 2020
Cited alongside, same era.
Big transfer (bit): General visual representation learning, 2020
Kolesnikov, A., Beyer, L., Zhai, X., Puigcerver, J., Yung, J., Gelly, S., and Houlsby, N · 2020
Cited alongside, same era.
Gshard: Scaling giant models with conditional computation and automatic sharding, 2020
Lepikhin, D., Lee, H., Xu, Y., Chen, D., Firat, O., Huang, Y., Krikun, M., Shazeer, N., and Chen, Z · 2020
Cited alongside, same era.
Multilingual denoising pre-training for neural machine translation, 2020
Liu, Y., Gu, J., Goyal, N., Li, X., Edunov, S., Ghazvininejad, M., Lewis, M., and Zettlemoyer, L · 2020
Cited alongside, same era.
Results of the WMT20 metrics shared task
Mathur, N., Wei, J., Freitag, M., Ma, Q., and Bojar, O · 2020
Cited alongside, same era.

Then

A neural scaling law from the dimension of the data manifold, 2020
Sharma, U. and Kaplan, J · 2020
Later among the works it cites.
Leveraging monolingual data with self-supervision for multilingual neural machine translation, 2020
Siddhant, A., Bapna, A., Cao, Y., Firat, O., Chen, M., Kudugunta, S., Arivazhagan, N., and Wu, Y · 2020
Later among the works it cites.
On layer normalization in the transformer architecture, 2020
Xiong, R., Yang, Y., He, D., Zheng, K., Zheng, S., Xing, C., Zhang, H., Lan, Y., Wang, L., and Liu, T.-Y · 2020
Later among the works it cites.
Recipes for safety in open-domain chatbots
Original
Xu, J., Ju, D., Li, M., Boureau, Y.-L., Weston, J., and Dinan, E · 2020
Later among the works it cites.
Scaling laws vs model architectures: How does inductive bias influence scaling? an extensive empirical study on language tasks, 2021
Anonymous · 2021
Later among the works it cites.
Explaining neural scaling laws
Original
Bahri, Y., Dyer, E., Kaplan, J., Lee, J., and Sharma, U · 2021
Later among the works it cites.
Experts, errors, and context: A large-scale study of human evaluation for machine translation, 2021
Freitag, M., Foster, G., Grangier, D., Ratnakar, V., Tan, Q., and Macherey, W · 2021
Later among the works it cites.
An empirical exploration in quality filtering of text data, 2021
Gao, L · 2021
Later among the works it cites.
Scaling laws for neural machine translation, 2021
Ghorbani, B., Firat, O., Freitag, M., Bapna, A., Krikun, M., Garcia, X., Chelba, C., and Cherry, C · 2021
Later among the works it cites.
Data and parameter scaling laws for neural machine translation
Gordon, M., Duh, K., and Kaplan, J · 2021
Later among the works it cites.
Learning curves for analysis of deep networks
Hoiem, D., Gupta, T., Li, Z., and Shlapentokh-Rothman, M · 2021
Later among the works it cites.
To ship or not to ship: An extensive evaluation of automatic metrics for machine translation, 2021
Kocmi, T., Federmann, C., Grundkiewicz, R., Junczys-Dowmunt, M., Matsushita, H., and Menezes, A · 2021
Later among the works it cites.
Accuracy on the line: on the strong correlation between out-of-distribution and in-distribution generalization
Miller, J. P., Taori, R., Raghunathan, A., Sagawa, S., Koh, P. W., Shankar, V., Liang, P., Carmon, Y., and Schmidt, L · 2021
Later among the works it cites.
Language models are good translators, 2021
Wang, S., Tu, Z., Tan, Z., Wang, W., Sun, M., and Liu, Y · 2021
Later among the works it cites.
Lamda: Language models for dialog applications, 2022
Thoppilan, R., Freitas, D. D., Hall, J., Shazeer, N., Kulshreshtha, A., Cheng, H.-T., Jin, A., Bos, T., Baker, L., Du, Y., Li, Y., Lee, H., Zheng, H. S., Ghafouri, A., Menegali, M., Huang, Y., Krikun, M., Lepikhin, D., Qin, J., Chen, D., Xu, Y., Chen, Z., Roberts, A., Bosma, M., Zhou, Y., Chang, C.-C., Krivokon, I., Rusch, W., Pickett, M., Meier-Hellstern, K., Morris, M. R., Doshi, T., Santos, R. D., Duke, T., Soraker, J., Zevenbergen, B., Prabhakaran, V., Diaz, M., Hutchinson, B., Olson, K., Molina, A., Hoffman-John, E., Lee, J., Aroyo, L., Rajakumar, R., Butryna, A., Lamm, M., Kuzmina, V., Fenton, J., Cohen, A., Bernstein, R., Kurzweil, R., Aguera-Arcas, B., Cui, C., Croak, M., Chi, E., and Le, Q · 2022
Closest in time.
Intelligent selection of language model training data
Moore, R. C. and Lewis, W · 2041
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…