Fetching the paper…

Reading the bibliography…

2024

Language Imbalance Driven Rewarding for Multilingual Self-improving

Yang, Wen, Wu, Junhong, Wang, Chen et al.

Original
PDF
Post on X

Understand

Large Language Models (LLMs) have achieved state-of-the-art performance across numerous tasks.

However, these advancements have predominantly benefited "first-class" languages such as English and Chinese, leaving many other languages underrepresented.
This imbalance, while limiting broader applications, generates a natural preference ranking between languages, offering an opportunity to bootstrap the multilingual capabilities of LLM in a self-improving manner.
Thus, we propose $\textit{Language Imbalance Driven Rewarding}$, where the inherent imbalance between dominant and non-dominant languages within LLMs is leveraged as a reward signal.

Language Imbalance Driven Rewarding for Multilingual Self-improving · Around

Built on

The independence of language and mathematical reasoning
Elizabeth M Brannon · 2005
Earlier work this paper cites.
Adam: A method for stochastic optimization
Original
Diederik P Kingma and Jimmy Ba · 2014
Earlier work this paper cites.
Think you have solved question answering? try arc, the ai2 reasoning challenge
Original
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord · 2018
Earlier work this paper cites.
Cross-lingual language model pretraining
Alexis Conneau and Guillaume Lample · 2019
Earlier work this paper cites.
Cross-lingual alignment of contextual word embeddings, with applications to zero-shot dependency parsing
Original
Tal Schuster, Ori Ram, Regina Barzilay, and Amir Globerson · 2019
Earlier work this paper cites.
Hellaswag: Can a machine really finish your sentence?
Original
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi · 2019
Earlier work this paper cites.
Measuring massive multitask language understanding
Original
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt · 2020
Earlier work this paper cites.
Training verifiers to solve math word problems
Original
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al · 2021
Earlier work this paper cites.
Truthfulqa: Measuring how models mimic human falsehoods
Original
Stephanie Lin, Jacob Hilton, and Owain Evans · 2021
Earlier work this paper cites.
Zero-infinity: Breaking the gpu memory wall for extreme scale deep learning
Samyam Rajbhandari, Olatunji Ruwase, Jeff Rasley, Shaden Smith, and Yuxiong He · 2021
Earlier work this paper cites.
Constitutional ai: Harmlessness from ai feedback
Original
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al · 2022
Earlier work this paper cites.
Accelerate: Training and inference at scale made simple, efficient and adaptable
Sylvain Gugger, Lysandre Debut, Thomas Wolf, Philipp Schmid, Zachary Mueller, Sourab Mangrulkar, Marc Sun, and Benjamin Bossan · 2022
Earlier work this paper cites.
Crosslingual generalization through multitask finetuning
Original
Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, et al · 2022
Earlier work this paper cites.
Training language models to follow instructions with human feedback
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al · 2022
Earlier work this paper cites.
Language models are multilingual chain-of-thought reasoners
Original
Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, et al · 2022
Earlier work this paper cites.
Self-instruct: Aligning language models with self-generated instructions
Original
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi · 2022
Earlier work this paper cites.
Gpt-4 technical report
Original
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al · 2023
Earlier work this paper cites.
Qwen technical report
Original
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al · 2023
Earlier work this paper cites.
Which prompts make the difference? data prioritization for efficient human llm evaluation
Original
Meriem Boubdir, Edward Kim, Beyza Ermis, Marzieh Fadaee, and Sara Hooker · 2023
Earlier work this paper cites.
Flashattention-2: Faster attention with better parallelism and work partitioning
Original
Tri Dao · 2023
Earlier work this paper cites.

Similar

Toxicity in chatgpt: Analyzing persona-assigned language models
Original
Ameet Deshpande, Vishvak Murahari, Tanmay Rajpurohit, Ashwin Kalyan, and Karthik Narasimhan · 2023
Cited alongside, same era.
Do multilingual language models think better in english?
Original
Julen Etxaniz, Gorka Azkune, Aitor Soroa, Oier Lopez de Lacalle, and Mikel Artetxe · 2023
Cited alongside, same era.
Are large language model-based evaluators the solution to scaling up multilingual evaluation?
Original
Rishav Hada, Varun Gumma, Adrian de Wynter, Harshita Diddee, Mohamed Ahmed, Monojit Choudhury, Kalika Bali, and Sunayana Sitaram · 2023
Cited alongside, same era.
Not all languages are created equal in llms: Improving multilingual capability by cross-lingual-thought prompting
Original
Haoyang Huang, Tianyi Tang, Dongdong Zhang, Wayne Xin Zhao, Ting Song, Yan Xia, and Furu Wei · 2023
Cited alongside, same era.

Then

Internlm2 technical report
Original
Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, et al · 2024
Closest in time.
xcot: Cross-lingual instruction tuning for cross-lingual chain-of-thought reasoning
Original
Linzheng Chai, Jian Yang, Tao Sun, Hongcheng Guo, Jiaheng Liu, Bing Wang, Xiannian Liang, Jiaqi Bai, Tongliang Li, Qiyao Peng, et al · 2024
Closest in time.
Rlhf deciphered: A critical analysis of reinforcement learning from human feedback for llms
Original
Shreyas Chaudhari, Pranjal Aggarwal, Vishvak Murahari, Tanmay Rajpurohit, Ashwin Kalyan, Karthik Narasimhan, Ameet Deshpande, and Bruno Castro da Silva · 2024
Closest in time.
Rlhf can speak many languages: Unlocking multilingual preference optimization for llms
Original
John Dang, Arash Ahmadian, Kelly Marchisio, Julia Kreutzer, Ahmet Üstün, and Sara Hooker · 2024
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al · 2023

Cited alongside, same era.

Efficient memory management for large language model serving with pagedattention

Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica · 2023

Cited alongside, same era.

Okapi: Instruction-tuned large language models in multiple languages with reinforcement learning from human feedback

Viet Dac Lai, Chien Van Nguyen, Nghia Trung Ngo, Thuat Nguyen, Franck Dernoncourt, Ryan A Rossi, and Thien Huu Nguyen · 2023

Cited alongside, same era.

Bloom: A 176b-parameter open-access multilingual language model

Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al · 2023

Cited alongside, same era.

G-eval: Nlg evaluation using gpt-4 with better human alignment

Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu · 2023

Cited alongside, same era.

Cross-lingual prompting: Improving zero-shot chain-of-thought reasoning across languages

Libo Qin, Qiguang Chen, Fuxuan Wei, Shijue Huang, and Wanxiang Che · 2023

Cited alongside, same era.

Does the english matter? elicit cross-lingual abilities of large language models

Leonardo Ranaldi and Giulia Pucci · 2023

Cited alongside, same era.

Branch-solve-merge improves large language model evaluation and generation

Swarnadeep Saha, Omer Levy, Asli Celikyilmaz, Mohit Bansal, Jason Weston, and Xian Li · 2023

Cited alongside, same era.

Negating negatives: Alignment with human negative samples via distributional dispreference optimization

Shitong Duan, Xiaoyuan Yi, Peng Zhang, Yan Liu, Zheng Liu, Tun Lu, Xing Xie, and Ning Gu · 2024

Closest in time.

Alpacafarm: A simulation framework for methods that learn from human feedback

Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy S Liang, and Tatsunori B Hashimoto · 2024

Closest in time.

A framework for few-shot language model evaluation, 07 2024

Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou · 2024

Closest in time.

Mitigating translationese with gpt-4: Strategies and performance

Maria Kunilovskaya, Koel Dutta Chowdhury, Heike Przybyl, Cristina España-Bonet, and Josef Genabith · 2024

Closest in time.

Introducing meta llama 3: The most capable openly available llm to date

AI Meta · 2024

Closest in time.

Iterative reasoning preference optimization

Richard Yuanzhe Pang, Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, and Jason Weston · 2024

Closest in time.

Multilingual large language model: A survey of resources, taxonomy and frontiers

Libo Qin, Qiguang Chen, Yuhang Zhou, Zhi Chen, Yinghui Li, Lizi Liao, Min Li, Wanxiang Che, and Philip S Yu · 2024

Closest in time.

Direct preference optimization: Your language model is secretly a reward model

Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn · 2024

Closest in time.

Mapo: Advancing multilingual reasoning through multilingual alignment-as-preference optimization

Shuaijie She, Shujian Huang, Wei Zou, Wenhao Zhu, Xiang Liu, Xiang Geng, and Jiajun Chen · 2024

Closest in time.

Gemma: Open models based on gemini research and technology

Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al · 2024

Closest in time.

Aya model: An instruction finetuned open-access multilingual language model

Ahmet Üstün, Viraat Aryabumi, Zheng-Xin Yong, Wei-Yin Ko, Daniel D’souza, Gbemileke Onilude, Neel Bhandari, Shivalika Singh, Hui-Lee Ooi, Amr Kayid, et al · 2024

Closest in time.

Qwen2 technical report

Original

An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al · 2024

Closest in time.

Self-rewarding language models

Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston · 2024

Closest in time.

Enhancing multilingual capabilities of large language models through self-distillation from resource-rich languages

Yuanchi Zhang, Yile Wang, Zijun Liu, Shuo Wang, Xiaolong Wang, Peng Li, Maosong Sun, and Yang Liu · 2024

Closest in time.

Llama beyond english: An empirical study on language capability transfer

Jun Zhao, Zhihao Zhang, Qi Zhang, Tao Gui, and Xuanjing Huang · 2024

Closest in time.