BPE Gets Picky: Efficient Vocabulary Refinement During Tokenizer Training

The meaning-frequency relationship of words

George Kingsley Zipf. 1945 · 1945

Earlier work this paper cites.

A new algorithm for data compression

Philip Gage. 1994 · 1994

Earlier work this paper cites.

Bleu: a method for automatic evaluation of machine translation

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002 · 2002

Earlier work this paper cites.

Statistical Significance Tests for Machine Translation Evaluation

Philipp Koehn. 2004 · 2004

Earlier work this paper cites.

Findings of the 2016 Conference on Machine Translation

Ondrej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno-Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Aurélie Névéol, Mariana Neves, Martin Popel, Matt Post, Raphaël Rubino, Carolina Scarton, Lucia Specia, Marco Turchi, Karin M. Verspoor, and Marcos Zampieri. 2016 · 2016

Earlier work this paper cites.

Neural Machine Translation of Rare Words with Subword Units

Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016 · 2016

Earlier work this paper cites.

Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation

Original

Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Łukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2016 · 2016

Earlier work this paper cites.

The University of Edinburgh’s neural MT systems for WMT17

Rico Sennrich, Alexandra Birch, Anna Currey, Ulrich Germann, Barry Haddow, Kenneth Heafield, Antonio Valerio Miceli Barone, and Philip Williams. 2017 · 2017

Earlier work this paper cites.

Subword Regularization: Improving Neural network Translation Models with Multiple Subword Candidates

Taku Kudo. 2018 · 2018

Earlier work this paper cites.

SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing

Taku Kudo and John Richardson. 2018 · 2018

Earlier work this paper cites.

A call for clarity in reporting BLEU scores

Matt Post. 2018 · 2018

Earlier work this paper cites.

Investigating the Effectiveness of BPE: The Power of Shorter Sequences

Matthias Gallé. 2019 · 2019

Earlier work this paper cites.

fairseq: A Fast, Extensible Toolkit for Sequence Modeling

Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019 · 2019

Earlier work this paper cites.

Revisiting low-resource neural machine translation: A case study

Rico Sennrich and Biao Zhang. 2019 · 2019

Earlier work this paper cites.

Byte Pair Encoding is Suboptimal for Language Model Pretraining

Kaj Bostrom and Greg Durrett. 2020 · 2020

Earlier work this paper cites.

BPE-dropout: Simple and effective subword regularization

Ivan Provilkov, Dmitrii Emelianenko, and Elena Voita. 2020 · 2020

Earlier work this paper cites.

COMET: A Neural Framework for MT evaluation

Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020 · 2020

Earlier work this paper cites.

Superbizarre is not superb: Derivational morphology improves BERT’s interpretation of complex words

Valentin Hofmann, Janet Pierrehumbert, and Hinrich Schütze. 2021 · 2021

Cited alongside, same era.

How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models

Phillip Rust, Jonas Pfeiffer, Ivan Vulić, Sebastian Ruder, and Iryna Gurevych. 2021 · 2021

Cited alongside, same era.

A statistical extension of byte-pair encoding

David Vilar and Marcello Federico. 2021 · 2021

Cited alongside, same era.

The Flores-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation

Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc’Aurelio Ranzato, Francisco Guzmán, and Angela Fan. 2022 · 2022

Cited alongside, same era.

An Embarrassingly Simple Method to Mitigate Undesirable Properties of Pretrained Language Model Tokenizers

Valentin Hofmann, Hinrich Schuetze, and Janet Pierrehumbert. 2022 · 2022

Cited alongside, same era.

An Analysis of BPE Vocabulary Trimming in Neural Machine Translation

Original

Marco Cognetta, Tatsuya Hiraoka, Naoaki Okazaki, Rico Sennrich, and Yuval Pinter. 2024 · 2024

Closest in time.

Coercing LLMs to do and reveal (almost) anything

Original

Jonas Geiping, Alex Stein, Manli Shu, Khalid Saifullah, Yuxin Wen, and Tom Goldstein. 2024 · 2024

Closest in time.

Unpacking Tokenization: Evaluating Text Compression and its Correlation with Model Performance

Original

Omer Goldman, Avi Caciularu, Matan Eyal, Kris Cao, Idan Szpektor, and Reut Tsarfaty. 2024 · 2024

Closest in time.

OLMo: Accelerating the Science of Language Models

Original

Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, et al. 2024 · 2024

Closest in time.