Domain-adaptative Continual Learning for Low-resource Tasks: Evaluation on Nepali

Energy and policy considerations for deep learning in nlp

Original

Emma Strubell, Ananya Ganesh, and Andrew McCallum. 2019 · 1906

Earlier work this paper cites.

Winogrande: An adversarial winograd schema challenge at scale

Original

Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2019 · 1907

Earlier work this paper cites.

Scaling laws for neural language models

Original

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020 · 2001

Earlier work this paper cites.

Language models are few-shot learners

Original

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020 · 2005

Earlier work this paper cites.

Measuring massive multitask language understanding

Original

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021 · 2009

Earlier work this paper cites.

Improving neural machine translation models with monolingual data

Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016 · 2016

Earlier work this paper cites.

Think you have solved question answering? try arc, the ai2 reasoning challenge

Original

Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018 · 2018

Earlier work this paper cites.

The FLORES evaluation datasets for low-resource machine translation: Nepali–English and Sinhala–English

Francisco Guzmán, Peng-Jen Chen, Myle Ott, Juan Pino, Guillaume Lample, Philipp Koehn, Vishrav Chaudhary, and Marc’Aurelio Ranzato. 2019 · 2019

Earlier work this paper cites.

Linguistic knowledge and transferability of contextual representations

Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. 2019 · 2019

Earlier work this paper cites.

A multiscale visualization of attention in the transformer model

Jesse Vig. 2019 · 2019

Earlier work this paper cites.

Analyzing the structure of attention in a transformer language model

Jesse Vig and Yonatan Belinkov. 2019 · 2019

Earlier work this paper cites.

Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned

Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. 2019 · 2019

Earlier work this paper cites.

Don’t stop pretraining: Adapt language models to domains and tasks

Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020 · 2020

Earlier work this paper cites.

Multi-stage pre-training for low-resource domain adaptation

Rong Zhang, Revanth Gangi Reddy, Md Arafat Sultan, Vittorio Castelli, Anthony Ferritto, Radu Florian, Efsun Sarioglu Kayi, Salim Roukos, Avi Sil, and Todd Ward. 2020 · 2020

Earlier work this paper cites.

On the dangers of stochastic parrots: Can language models be too big?

Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021 · 2021

Earlier work this paper cites.

A survey of data augmentation approaches for NLP

Steven Y. Feng, Varun Gangal, Jason Wei, Sarath Chandar, Soroush Vosoughi, Teruko Mitamura, and Eduard Hovy. 2021 · 2021

Earlier work this paper cites.

Lora: Low-rank adaptation of large language models

Original

Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021 · 2021

Cited alongside, same era.

mDAPT: Multilingual domain adaptive pretraining in a single model

Rasmus Kær Jørgensen, Mareike Hartmann, Xiang Dai, and Desmond Elliott. 2021 · 2021

Cited alongside, same era.

Carbon emissions and large neural network training

Original

David A. Patterson, Joseph Gonzalez, Quoc V. Le, Chen Liang, Lluís-Miquel Munguía, Daniel Rothchild, David R. So, Maud Texier, and Jeff Dean. 2021 · 2021

Cited alongside, same era.

Towards a Cleaner Document-Oriented Multilingual Crawled Corpus

Original

Julien Abadji, Pedro Ortiz Suarez, Laurent Romary, and Benoît Sagot. 2022 · 2022

Cited alongside, same era.

Computational historical linguistics and language diversity in South Asia

Openhathi series: An approach to build bilingual llms frugally

SarvamAI. 2023 · 2023

Later among the works it cites.

Stanford alpaca: An instruction-following llama model

Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023 · 2023

Later among the works it cites.

To code, or not to code? exploring impact of code in pre-training

Original

Viraat Aryabumi, Yixuan Su, Raymond Ma, Adrien Morisot, Ivan Zhang, Acyr Locatelli, Marzieh Fadaee, Ahmet Üstün, and Sara Hooker. 2024 · 2024

Closest in time.

Lora learns less and forgets less

Original

Dan Biderman, Jacob Portes, Jose Javier Gonzalez Ortiz, Mansheej Paul, Philip Greengard, Connor Jennings, Daniel King, Sam Havens, Vitaliy Chiley, Jonathan Frankle, Cody Blakeney, and John P. Cunningham. 2024 · 2024

Closest in time.

The llama 3 herd of models

Original