Investigating Mysteries of CoT-Augmented Distillation

Distributed representations of words and phrases and their compositionality

Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeffrey Dean. 2013 · 2013

Earlier work this paper cites.

Distilling the knowledge in a neural network

Original

Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015 · 2015

Earlier work this paper cites.

Axiomatic attribution for deep networks

Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017 · 2017

Earlier work this paper cites.

Can a suit of armor conduct electricity? a new dataset for open book question answering

Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018 · 2018

Earlier work this paper cites.

Quarel: A dataset and models for answering questions about qualitative relationships

Oyvind Tafjord, Peter Clark, Matt Gardner, Wen tau Yih, and Ashish Sabharwal. 2018 · 2018

Earlier work this paper cites.

Language models are unsupervised multitask learners

Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019 · 2019

Earlier work this paper cites.

CommonsenseQA: A question answering challenge targeting commonsense knowledge

Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019 · 2019

Earlier work this paper cites.

Preemptive answer "attacks" on chain-of-thought reasoning

Original

Rongwu Xu, Zehan Qi, and Wei Xu. 2024 · 2019

Earlier work this paper cites.

interpreting gpt: the logit lens

nostalgebraist. 2020 · 2020

Earlier work this paper cites.

Transformers: State-of-the-art natural language processing

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020 · 2020

Earlier work this paper cites.

When can models learn from explanations? a formal framework for understanding the roles of explanation data

Peter Hase and Mohit Bansal. 2022 · 2022

Cited alongside, same era.

Symbolic knowledge distillation: from general language models to commonsense models

Peter West, Chandra Bhagavatula, Jack Hessel, Jena Hwang, Liwei Jiang, Ronan Le Bras, Ximing Lu, Sean Welleck, and Yejin Choi. 2022 · 2022

Cited alongside, same era.

Reframing human-AI collaboration for generating free-text explanations

Sarah Wiegreffe, Jack Hessel, Swabha Swayamdipta, Mark Riedl, and Yejin Choi. 2022 · 2022

Cited alongside, same era.

Eliciting latent predictions from transformers with the tuned lens

Original

Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Jacob Steinhardt. 2023 · 2023

Cited alongside, same era.

Implicit chain of thought reasoning via knowledge distillation

Original

Teaching small language models to reason

Original

Lucie Charlotte Magister, Jonathan Mallinson, Jakub Adamek, Eric Malmi, and Aliaksei Severyn. 2023 · 2023

Later among the works it cites.

Future lens: Anticipating subsequent tokens from a single hidden state

Koyena Pal, Jiuding Sun, Andrew Yuan, Byron Wallace, and David Bau. 2023 · 2023

Later among the works it cites.

Distilling reasoning capabilities into smaller language models

Kumar Shridhar, Alessandro Stolfo, and Mrinmaya Sachan. 2023 · 2023

Later among the works it cites.

Revisiting relation extraction in the era of large language models

Somin Wadhwa, Silvio Amir, and Byron Wallace. 2023 · 2023

Later among the works it cites.

SCOTT: Self-consistent chain-of-thought distillation

Peifeng Wang, Zhengyang Wang, Zheng Li, Yifan Gao, Bing Yin, and Xiang Ren. 2023 · 2023

Later among the works it cites.