Fetching the paper…

Reading the bibliography…

2023

LLMZip: Lossless Text Compression using Large Language Models

Valmeekam, Chandra Shekhara Kaushik, Narayanan, Krishna, Kalathil, Dileep et al.

Original
PDF
Post on X

Understand

We provide new estimates of an asymptotic upper bound on the entropy of English using the large language model LLaMA-7B as a predictor for the next token given a window of past tokens.

This estimate is significantly smaller than currently available estimates in \cite{cover1978convergent}, \cite{lutati2023focus}.
A natural byproduct is an algorithm for lossless compression of English text which combines the prediction from the large language model with a lossless compression scheme.
Preliminary results from limited experiments suggest that our scheme outperforms state-of-the-art text compression schemes such as BSC, ZPAQ, and paq8h.

LLMZip: Lossless Text Compression using Large Language Models · Around

Built on

“Prediction and entropy of printed english,”
Claude E Shannon, · 1951
Earlier work this paper cites.
“A convergent gambling estimate of the entropy of english,”
Thomas Cover and Roger King, · 1978
Earlier work this paper cites.
“Data compression using adaptive coding and partial string matching,”
John Cleary and Ian Witten, · 1984
Earlier work this paper cites.
“Modeling for text compression,”
Timothy Bell, Ian H Witten, and John G Cleary, · 1989
Earlier work this paper cites.

Similar

Elements of Information Theory
Thomas M Cover and Joy A Thomas, · 1999
Cited alongside, same era.
Information theory, inference and learning algorithms
David JC MacKay, · 2003
Cited alongside, same era.
“Deepzip: Lossless data compression using recurrent neural networks,”
Original
Mohit Goyal, Kedar Tatwawadi, Shubham Chandak, and Idoia Ochoa, · 2018
Cited alongside, same era.
“Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing,”
Original
Taku Kudo and John Richardson, · 2018
Cited alongside, same era.
“text8 results,” http://mattmahoney.net/dc/textdata.html
Cited in the paper.

Then

“Focus your attention (with adaptive IIR filters),” 2023
Shahar Lutati, Itamar Zimerman, and Lior Wolf, · 2023
Closest in time.
“Llama: Open and efficient foundation language models,” 2023
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample, · 2023
Closest in time.
Legends of Texas
J. Frank Dobie, · 2023
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…