Two Counterexamples to Tokenization and the Noiseless Channel

A mathematical theory of communication

Claude E. Shannon. 1948 · 1948

Earlier work this paper cites.

A new algorithm for data compression

Philip Gage. 1994 · 1994

Earlier work this paper cites.

Joint tokenization and translation

Xinyan Xiao, Yang Liu, Young-Sook Hwang, Qun Liu, and Shouxun Lin. 2010 · 2010

Earlier work this paper cites.

Neural machine translation of rare words with subword units

Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016 · 2016

Earlier work this paper cites.

Subword regularization: Improving neural network translation models with multiple subword candidates

Taku Kudo. 2018 · 2018

Cited alongside, same era.

How much does tokenization affect neural machine translation?

Miguel Domingo, Mercedes García-Martínez, Alexandre Helle, Francisco Casacuberta, and Manuel Herranz. 2019 · 2019

Cited alongside, same era.

Finding the optimal vocabulary size for neural machine translation

Thamme Gowda and Jonathan May. 2020 · 2020

Cited alongside, same era.

Dynamic programming encoding for subword segmentation in neural machine translation

Xuanli He, Gholamreza Haffari, and Mohammad Norouzi. 2020 · 2020

Cited alongside, same era.

Tokenization and the noiseless channel

Vilém Zouhar, Clara Meister, Juan Gastaldi, Li Du, Mrinmaya Sachan, and Ryan Cotterell. 2023a

Cited in the paper.

BPE-dropout: Simple and effective subword regularization

Ivan Provilkov, Dmitrii Emelianenko, and Elena Voita. 2020 · 2020

Later among the works it cites.

Joint optimization of tokenization and downstream model

Tatsuya Hiraoka, Sho Takase, Kei Uchiumi, Atsushi Keyaki, and Naoaki Okazaki. 2021 · 2021

Later among the works it cites.

A formal perspective on byte-pair encoding

Vilém Zouhar, Clara Meister, Juan Gastaldi, Li Du, Tim Vieira, Mrinmaya Sachan, and Ryan Cotterell. 2023b · 2023

Later among the works it cites.