Unused information in token probability distribution of generative LLM: improving LLM reading comprehension through calculation of expected values

A. Holtzman, J. Buys, L. Du, M. Forbes, and Y. Choi, “The curious case of neural text degeneration,” in 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 . OpenReview.net, 2020. doi: 10.48550/arXiv.1904.09751. [Online]. Available: https://openreview.net/forum?id=rygGQyrFvH

1904

Earlier work this paper cites.

H. P. Grice, “Logic and conversation,” in Syntax and Semantics: Vol. 3: Speech Acts , P. Cole and J. L. Morgan, Eds. New York: Academic Press, 1975. doi: 10.1163/9789004368811_003 pp. 41–58

1975

Earlier work this paper cites.

S. Basu, G. S. Ramachandran, N. S. Keskar, and L. R. Varshney, “{MIROSTAT}: A {neural} {text} {decoding} {algorithm} {that} {directly} {controls} {perplexity},” in International Conference on Learning Representations , 2021. doi: 10.48550/arXiv.2007.14966

2007

Earlier work this paper cites.

Z. Yang, Z. Dai, R. Salakhutdinov, and W. W. Cohen, “Breaking the softmax bottleneck: A high-rank rnn language model,” 2018. doi: 10.48550/arXiv.1711.03953

2018

Earlier work this paper cites.

T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y. Jernite, J. Plu, C. Xu, T. Le Scao, S. Gugger, M. Drame, Q. Lhoest, and A. Rush, “Transformers: State-of-the-art natural language processing,” in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , Q. Liu and D. Schlangen, Eds. Online: Association for Computational Linguistics, Oct. 2020. doi: 10.18653/v1/2020.emnlp-demos.6 pp. 38–45. [Online]. Available: https://aclanthology.org/2020.emnlp-demos.6

2020

Earlier work this paper cites.

A. R. Fabbri, W. Kryściński, B. McCann, C. Xiong, R. Socher, and D. Radev, “SummEval: Re-evaluating summarization evaluation,” Transactions of the Association for Computational Linguistics , vol. 9, 2021. doi: 10.1162/tacl_a_00373. [Online]. Available: https://aclanthology.org/2021.tacl-1.24

2021

Earlier work this paper cites.

C. Meister, M. Forster, and R. Cotterell, “Determinantal beam search,” in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , C. Zong, F. Xia, W. Li, and R. Navigli, Eds. Online: Association for Computational Linguistics, Aug. 2021. doi: 10.18653/v1/2021.acl-long.512 pp. 6551–6562. [Online]. Available: https://aclanthology.org/2021.acl-long.512

2021

Earlier work this paper cites.

M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba, “Evaluating large language models trained on code,” 2021. doi: 10.48550/arXiv.2107.03374

2021

Earlier work this paper cites.

J. Wei, Y. Tay, R. Bommasani, C. Raffel, B. Zoph, S. Borgeaud, D. Yogatama, M. Bosma, D. Zhou, D. Metzler, E. H. Chi, T. Hashimoto, O. Vinyals, P. Liang, J. Dean, and W. Fedus, “Emergent abilities of large language models,” 2022. doi: 10.48550/arXiv.2206.07682

2022

Cited alongside, same era.

C. Meister, T. Pimentel, G. Wiher, and R. Cotterell, “Locally typical sampling,” Transactions of the Association for Computational Linguistics , vol. 11, pp. 102–121, 2022. doi: 10.48550/arXiv.2202.00666. [Online]. Available: https://api.semanticscholar.org/CorpusID:252918666

2022

Cited alongside, same era.

L. Gao, J. Tow, B. Abbasi, S. Biderman, S. Black, A. DiPofi, C. Foster, L. Golding, J. Hsu, A. Le Noac’h, H. Li, K. McDonell, N. Muennighoff, C. Ociepa, J. Phang, L. Reynolds, H. Schoelkopf, A. Skowron, L. Sutawika, E. Tang, A. Thite, B. Wang, K. Wang, and A. Zou, “A framework for few-shot language model evaluation,” 12 2023. doi: 10.5281/zenodo.10256836. [Online]. Available: https://zenodo.org/records/10256836

2023

Cited alongside, same era.

W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. Gonzalez, H. Zhang, and I. Stoica, “Efficient memory management for large language model serving with pagedattention,” in Proceedings of the 29th Symposium on Operating Systems Principles , ser. SOSP ’23. New York, NY, USA: Association for Computing Machinery, 2023. doi: 10.1145/3600006.3613165. ISBN 9798400702297 p. 611–626. [Online]. Available: https://doi.org/10.1145/3600006.3613165

2023

Later among the works it cites.

C. Shen, L. Cheng, X.-P. Nguyen, Y. You, and L. Bing, “Large language models are not yet human-level evaluators for abstractive summarization,” in Findings of the Association for Computational Linguistics: EMNLP 2023 , H. Bouamor, J. Pino, and K. Bali, Eds. Singapore: Association for Computational Linguistics, Dec. 2023. doi: 10.18653/v1/2023.findings-emnlp.278 pp. 4215–4233. [Online]. Available: https://aclanthology.org/2023.findings-emnlp.278

2023

Later among the works it cites.

L. Vilnis, Y. Zemlyanskiy, P. Murray, A. Passos, and S. Sanghai, “Arithmetic sampling: parallel diverse decoding for large language models,” in Proceedings of the 40th International Conference on Machine Learning , ser. ICML’23. JMLR.org, 2023. doi: 10.5555/3618408.3619870