FuxiTranyu: A Multilingual Large Language Model Trained with Balanced Data

Language models are few-shot learners

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020 · 1901

Earlier work this paper cites.

Megatron-LM: Training multi-billion parameter language models using model parallelism

Original

Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019 · 1909

Earlier work this paper cites.

Unsupervised cross-lingual representation learning at scale

Original

Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019 · 1911

Earlier work this paper cites.

GLU variants improve transformer

Original

Noam Shazeer. 2020 · 2002

Earlier work this paper cites.

Measuring massive multitask language understanding

Original

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020 · 2009

Earlier work this paper cites.

MultiUN: A multilingual corpus from united nation documents

Andreas Eisele and Yu Chen. 2010 · 2010

Earlier work this paper cites.

KenLM: Faster and smaller language model queries

Kenneth Heafield. 2011 · 2011

Earlier work this paper cites.

MultiUN v2: UN documents with multilingual alignments

Yu Chen and Andreas Eisele. 2012 · 2012

Earlier work this paper cites.

Findings of the 2014 workshop on statistical machine translation

Ondrej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, Radu Soricut, Lucia Specia, and Ale s Tamchyna. 2014 · 2014

Earlier work this paper cites.

Findings of the 2016 conference on machine translation

Ond rej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Aurelie Neveol, Mariana Neves, Martin Popel, Matt Post, Raphael Rubino, Carolina Scarton, Lucia Specia, Marco Turchi, Karin Verspoor, and Marcos Zampieri. 2016 · 2016

Earlier work this paper cites.

Bridging nonlinearities and stochastic regularizers with Gaussian error linear units

Original

Dan Hendrycks and Kevin Gimpel. 2016 · 2016

Earlier work this paper cites.

OpenSubtitles2016: Extracting large parallel corpora from movie and TV subtitles

Pierre Lison and Jörg Tiedemann. 2016 · 2016

Earlier work this paper cites.

Overview of the IWSLT 2017 evaluation campaign

Mauro Cettolo, Marcello Federico, Luisa Bentivogli, Jan Niehues, Sebastian Stüker, Katsuhito Sudoh, Koichiro Yoshino, and Christian Federmann. 2017 · 2017

Earlier work this paper cites.

Decoupled weight decay regularization

Original

Ilya Loshchilov and Frank Hutter. 2017 · 2017

Earlier work this paper cites.

Proximal policy optimization algorithms

Original

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017 · 2017

Earlier work this paper cites.

Think you have solved question answering? try arc, the AI2 reasoning challenge

Original

Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018 · 2018

Earlier work this paper cites.

Asynchronous pipeline for processing huge corpora on medium to low resource infrastructures

Pedro Javier Ortiz Suárez, Benoît Sagot, and Laurent Romary. 2019 · 2019

Earlier work this paper cites.

HellaSwag: Can a machine really finish your sentence?

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019 · 2019

Earlier work this paper cites.

Root mean square layer normalization

Biao Zhang and Rico Sennrich. 2019 · 2019

Earlier work this paper cites.

Emerging cross-lingual structure in pretrained language models

Alexis Conneau, Shijie Wu, Haoran Li, Luke Zettlemoyer, and Veselin Stoyanov. 2020 · 2020

Earlier work this paper cites.

S2ORC: The semantic scholar open research corpus

Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kinney, and Daniel Weld. 2020 · 2020

Earlier work this paper cites.

A monolingual approach to contextualized word embeddings for mid-resource languages

Pedro Javier Ortiz Su’arez, Laurent Romary, and Benoit Sagot. 2020 · 2020

Earlier work this paper cites.

XCOPA: A multilingual dataset for causal commonsense reasoning

Edoardo Maria Ponti, Goran Glavaš, Olga Majewska, Qianchu Liu, Ivan Vulić, and Anna Korhonen. 2020 · 2020

Earlier work this paper cites.

Exploring the limits of transfer learning with a unified text-to-text transformer

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020 · 2020

Earlier work this paper cites.

Zero: Memory optimizations toward training trillion parameter models

Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2020 · 2020

Earlier work this paper cites.

On layer normalization in the transformer architecture

Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu. 2020 · 2020

Earlier work this paper cites.

XL-sum: Large-scale multilingual abstractive summarization for 44 languages

Tahmid Hasan, Abhik Bhattacharjee, Md. Saiful Islam, Kazi Mubasshir, Yuan-Fang Li, Yong-Bin Kang, M. Sohel Rahman, and Rifat Shahriyar. 2021 · 2021

Earlier work this paper cites.

Few-shot learning with multilingual language models

Original

Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O’Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona T. Diab, Veselin Stoyanov, and Xian Li. 2021 · 2021

Earlier work this paper cites.

Roformer: Enhanced transformer with rotary position embedding

Original

Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. 2021 · 2021

Earlier work this paper cites.

It’s all in the heads: Using attention heads as a baseline for cross-lingual transfer in commonsense reasoning

Original

Alexey Tikhonov and Max Ryabinin. 2021 · 2021

Cited alongside, same era.

Language-agnostic representation from multilingual sentence encoders for cross-lingual similarity estimation

Nattapong Tiyajamorn, Tomoyuki Kajiwara, Yuki Arase, and Makoto Onizuka. 2021 · 2021

Cited alongside, same era.

Ernie-code: Beyond english-centric cross-lingual pretraining for programming languages

Original

Yekun Chai, Shuohuan Wang, Chao Pang, Yu Sun, Hao Tian, and Hua Wu. 2022 · 2022

Cited alongside, same era.

The geometry of multilingual language model representations

Tyler A. Chang, Zhuowen Tu, and Benjamin K. Bergen. 2022 · 2022

Cited alongside, same era.

PaLM: Scaling language modeling with pathways

Original

Openhermes 2.5: An open dataset of synthetic data for generalist llm assistants

Teknium. 2023 · 2023

Later among the works it cites.

Zephyr: Direct distillation of lm alignment

Original

Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, et al. 2023 · 2023

Later among the works it cites.

Polylm: An open source polyglot large language model

Original

Xiangpeng Wei, Haoran Wei, Huan Lin, Tianhao Li, Pei Zhang, Xingzhang Ren, Mei Li, Yu Wan, Zhiwei Cao, Binbin Xie, et al. 2023 · 2023

Later among the works it cites.

DEPN: detecting and editing privacy neurons in pretrained language models

Xinwei Wu, Junzhuo Li, Minghui Xu, Weilong Dong, Shuangzhi Wu, Chao Bian, and Deyi Xiong. 2023 · 2023

Later among the works it cites.

Language representation projection: Can we transfer factual knowledge across languages in multilingual language models?