Fetching the paper…
Reading the bibliography…
When generating natural language from neural probabilistic models, high probability does not always coincide with high quality: It has often been observed that mode-seeking decoding methods, i.e., those that produce high-probability text under the model, lead to unnatural language.
Language models are few-shot learners
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020 · 1901
Earlier work this paper cites.
A mathematical theory of communication
Claude E. Shannon. 1948 · 1948
Earlier work this paper cites.
Human Behavior and the Principle of Least Effort
George Kingsley Zipf. 1949 · 1949
Earlier work this paper cites.
The origin of speech
Charles F. Hockett. 1960 · 1960
Earlier work this paper cites.
A probabilistic Earley parser as a psycholinguistic model
John T. Hale. 2001 · 2001
Earlier work this paper cites.
Efficiency and Complexity in Grammars
John A. Hawkins. 2004 · 2004
Earlier work this paper cites.
Word lengths are optimized for efficient communication
Steven T. Piantadosi, Harry Tily, and Edward Gibson. 2011 · 2011
Earlier work this paper cites.
The effect of word predictability on reading time is logarithmic
Nathaniel J. Smith and Roger Levy. 2013 · 2013
Earlier work this paper cites.
Fitting sentence level translation evaluation with many dense features
Miloš Stanojević and Khalil Sima’an. 2014 · 2014
Earlier work this paper cites.
Abstractive text summarization using sequence-to-sequence RNNs and beyond
Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Çağlar Gulçehre, and Bing Xiang. 2016 · 2016
Earlier work this paper cites.
Diverse beam search: Decoding diverse solutions from neural sequence models
Ashwin K. Vijayakumar, Michael Cogswell, Ramprasaath R. Selvaraju, Qing Sun, Stefan Lee, David J. Crandall, and Dhruv Batra. 2016 · 2016
Earlier work this paper cites.
Hierarchical Neural Story Generation
Angela Fan, Mike Lewis, and Yann Dauphin. 2018 · 2018
Earlier work this paper cites.
Different languages, similar encoding efficiency: Comparable information rates across the human communicative niche
Christophe Coupé, Yoon Mi Oh, Dan Dediu, and François Pellegrino. 2019 · 2019
Cited alongside, same era.
How efficiency shapes human language
Edward Gibson, Richard Futrell, Steven T. Piantadosi, Isabelle Dautriche, Kyle Mahowald, Leon Bergen, and Roger Levy. 2019 · 2019
Cited alongside, same era.
Comparison of Diverse Decoding Methods from Conditional Language Models
Daphne Ippolito, Reno Kriz, João Sedoc, Maria Kustikova, and Chris Callison-Burch. 2019 · 2019
Cited alongside, same era.
Language models are unsupervised multitask learners
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019 · 2019
Cited alongside, same era.
On NMT search errors and model errors: Cat got your tongue?
Felix Stahlberg and Bill Byrne. 2019 · 2019
Cited alongside, same era.
Musings on typicality
Transformers: State-of-the-art natural language processing
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020 · 2020
Later among the works it cites.
Mirostat: A perplexity-controlled neural text decoding algorithm
Sourya Basu, Govardana Sachitanandam Ramachandran, Nitish Shirish Keskar, and Lav R. Varshney. 2021 · 2021
Later among the works it cites.
All that’s ‘human’ is not gold: Evaluating human evaluation of generated text
Elizabeth Clark, Tal August, Sofia Serrano, Nikita Haduong, Suchin Gururangan, and Noah A. Smith. 2021 · 2021
Later among the works it cites.
Decoding methods for neural narrative generation
Alexandra DeLucia, Aaron Mueller, Xiang Lisa Li, and João Sedoc. 2021 · 2021
Later among the works it cites.
A surprisal–duration trade-off across and within the world’s languages
Tiago Pimentel, Clara Meister, Elizabeth Salesky, Simone Teufel, Damián Blasi, and Ryan Cotterell. 2021a · 2021
Later among the works it cites.
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Sander Dieleman. 2020 · 2020
Cited alongside, same era.
Is MAP decoding all you need? The inadequacy of the mode in neural machine translation
Bryan Eikema and Wilker Aziz. 2020 · 2020
Cited alongside, same era.
The curious case of neural text degeneration
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020 · 2020
Cited alongside, same era.
BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020 · 2020
Cited alongside, same era.
If beam search is the answer, what was the question?
Clara Meister, Ryan Cotterell, and Tim Vieira. 2020 · 2020
Cited alongside, same era.
Consistency of a recurrent language model with respect to incomplete decoding
Sean Welleck, Ilia Kulikov, Jaedeok Kim, Richard Yuanzhe Pang, and Kyunghyun Cho. 2020 · 2020
Cited alongside, same era.
On the predictive power of neural language models for human real-time comprehension behavior
Ethan Gotlieb Wilcox, Jon Gauthier, Jennifer Hu, Peng Qian, and Roger Levy. 2020 · 2020
Cited alongside, same era.
How (non-)optimal is the lexicon?
Tiago Pimentel, Irene Nikkarinen, Kyle Mahowald, Ryan Cotterell, and Damián Blasi. 2021b · 2021
Later among the works it cites.
Human evaluation of automatically generated text: Current trends and best practice guidelines
Chris van der Lee, Albert Gatt, Emiel van Miltenburg, and Emiel Krahmer. 2021 · 2021
Later among the works it cites.
A cognitive regularizer for language modeling
Jason Wei, Clara Meister, and Ryan Cotterell. 2021 · 2021
Later among the works it cites.
Trading off diversity and quality in natural language generation
Hugh Zhang, Daniel Duckworth, Daphne Ippolito, and Arvind Neelakantan. 2021 · 2021
Later among the works it cites.
Typical decoding for natural language generation
Clara Meister, Tiago Pimentel, Gian Wiher, and Ryan Cotterell. 2022 · 2022
Closest in time.
On decoding strategies for neural text generators
Gian Wiher, Clara Meister, and Ryan Cotterell. 2022 · 2022
Closest in time.