Honey, I Shrunk the Language Model: Impact of Knowledge Distillation Methods on Performance and Explainability

R. Tang, Y. Lu, L. Liu, L. Mou, O. Vechtomova, and J. Lin, “Distilling Task-Specific Knowledge from BERT into Simple Neural Networks,” Mar. 2019, arXiv:1903.12136 [cs]. [Online]. Available: http://arxiv.org/abs/1903.12136

Original

1903

Earlier work this paper cites.

Y. Feng, H. Wang, D. T. Yi, and R. Hu, “Triplet Distillation for Deep Face Recognition,” May 2019, arXiv:1905.04457 [cs]. [Online]. Available: http://arxiv.org/abs/1905.04457

Original

1905

Earlier work this paper cites.

D. Kaushik, E. Hovy, and Z. C. Lipton, “Learning the Difference that Makes a Difference with Counterfactually-Augmented Data,” Feb. 2020, arXiv:1909.12434 [cs, stat]. [Online]. Available: http://arxiv.org/abs/1909.12434

Original

1909

Earlier work this paper cites.

A. Vargha, H. D. Delaney, and A. Vargha, “A Critique and Improvement of the ”CL” Common Language Effect Size Statistics of McGraw and Wong,” Journal of Educational and Behavioral Statistics , vol. 25, no. 2, p. 101, 2000. [Online]. Available: http://links.jstor.org/sici?sici=1076-9986%28200022%2925%3A2%3C101%3AACAIOT%3E2.0.CO%3B2-O&origin=crossref

2000

Earlier work this paper cites.

H. Wang, Y. Li, Y. Wang, H. Hu, and M.-H. Yang, “Collaborative Distillation for Ultra-Resolution Universal Style Transfer,” Mar. 2020, arXiv:2003.08436 [cs, eess]. [Online]. Available: http://arxiv.org/abs/2003.08436

Original

2003

Earlier work this paper cites.

H. O. Mayer, Interview und schriftliche Befragung: Entwicklung, Durchführung und Auswertung , 4th ed., ser. 150 Jahre Wissen für die Zukunft. München Wien: Oldenbourg, 2008

2008

Earlier work this paper cites.

G. Norman, “Likert scales, levels of measurement and the “laws” of statistics,” Advances in Health Sciences Education , vol. 15, no. 5, pp. 625–632, Dec. 2010. [Online]. Available: http://link.springer.com/10.1007/s10459-010-9222-y

2010

Earlier work this paper cites.

A. Bhattacherjee, Social Science Research: Principles, Methods and Practices . Open Textbook Library, 01 2012

2012

Earlier work this paper cites.

G. Hinton, O. Vinyals, and J. Dean, “Distilling the Knowledge in a Neural Network,” Mar. 2015, arXiv:1503.02531 [cs, stat]. [Online]. Available: http://arxiv.org/abs/1503.02531

Original

2015

Earlier work this paper cites.

A. Dinno, “Nonparametric Pairwise Multiple Comparisons in Independent Groups using Dunn’s Test,” The Stata Journal: Promoting communications on statistics and Stata , vol. 15, no. 1, pp. 292–300, Apr. 2015. [Online]. Available: http://journals.sagepub.com/doi/10.1177/1536867X1501500117

2015

Earlier work this paper cites.

L. A. Hendricks, Z. Akata, M. Rohrbach, J. Donahue, B. Schiele, and T. Darrell, “Generating visual explanations,” in Computer Vision – ECCV 2016 , B. Leibe, J. Matas, N. Sebe, and M. Welling, Eds. Cham: Springer International Publishing, 2016, pp. 3–19

2016

Earlier work this paper cites.

F. Doshi-Velez and B. Kim, “Towards a rigorous science of interpretable machine learning,” arXiv: Machine Learning , 2017. [Online]. Available: https://api.semanticscholar.org/CorpusID:11319376

2017

Earlier work this paper cites.

G. Chen, W. Choi, X. Yu, T. Han, and M. Chandraker, “Learning Efficient Object Detection Models with Knowledge Distillation,” in Advances in Neural Information Processing Systems , vol. 30. Curran Associates, Inc., 2017. [Online]. Available: https://papers.nips.cc/paper_files/paper/2017/hash/e1e32e235eee1f970470a3a6658dfdd5-Abstract.html

2017

Earlier work this paper cites.

J. Kim, A. Rohrbach, T. Darrell, J. Canny, and Z. Akata, “Textual explanations for self-driving vehicles,” in Computer Vision – ECCV 2018 , V. Ferrari, M. Hebert, C. Sminchisescu, and Y. Weiss, Eds. Cham: Springer International Publishing, 2018, pp. 577–593

2018

Earlier work this paper cites.

D. H. Park, L. A. Hendricks, Z. Akata, A. Rohrbach, B. Schiele, T. Darrell, and M. Rohrbach, “Multimodal explanations: Justifying decisions and pointing to the evidence,” 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 8779–8788, 2018. [Online]. Available: https://api.semanticscholar.org/CorpusID:3604848

2018

Earlier work this paper cites.

A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever et al. , “Language models are unsupervised multitask learners,” OpenAI blog , vol. 1, no. 8, p. 9, 2019

2019

Earlier work this paper cites.

P. Izsak, S. Guskin, and M. Wasserblat, “Training Compact Models for Low Resource Entity Tagging using Pre-trained Language Models,” in 2019 Fifth Workshop on Energy Efficient Machine Learning and Cognitive Computing - NeurIPS Edition (EMC2-NIPS) , Dec. 2019, pp. 44–47. [Online]. Available: https://ieeexplore.ieee.org/document/9463575#page=1.73

2019

Earlier work this paper cites.

A. Holzinger, G. Langs, H. Denk, K. Zatloukal, and H. Müller, “Causability and explainability of artificial intelligence in medicine,” Wiley Interdisciplinary Reviews. Data Mining and Knowledge Discovery , vol. 9, 2019. [Online]. Available: https://api.semanticscholar.org/CorpusID:132372213

2019

Earlier work this paper cites.

N. F. Rajani, B. McCann, C. Xiong, and R. Socher, “Explain Yourself! Leveraging Language Models for Commonsense Reasoning,” in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics . Florence, Italy: Association for Computational Linguistics, 2019, pp. 4932–4942. [Online]. Available: https://www.aclweb.org/anthology/P19-1487

2019

Earlier work this paper cites.

A. Talmor, J. Herzig, N. Lourie, and J. Berant, “CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge,” in Proceedings of the 2019 Conference of the North . Minneapolis, Minnesota: Association for Computational Linguistics, 2019, pp. 4149–4158. [Online]. Available: http://aclweb.org/anthology/N19-1421

2019

Earlier work this paper cites.

T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei, “Language models are few-shot learners,” in Proceedings of the 34th International Conference on Neural Information Processing Systems , ser. NIPS ’20. Red Hook, NY, USA: Curran Associates Inc., 2020

2020

Earlier work this paper cites.

X. Jiao, Y. Yin, L. Shang, X. Jiang, X. Chen, L. Li, F. Wang, and Q. Liu, “TinyBERT: Distilling BERT for Natural Language Understanding,” in Findings of the Association for Computational Linguistics: EMNLP 2020 , T. Cohn, Y. He, and Y. Liu, Eds. Online: Association for Computational Linguistics, Nov. 2020, pp. 4163–4174. [Online]. Available: https://aclanthology.org/2020.findings-emnlp.372/

2020

Earlier work this paper cites.

E. Strubell, A. Ganesh, and A. McCallum, “Energy and policy considerations for modern deep learning research,” Proceedings of the AAAI Conference on Artificial Intelligence , vol. 34, no. 09, pp. 13 693–13 696, Apr. 2020. [Online]. Available: https://ojs.aaai.org/index.php/AAAI/article/view/7123

2020

Earlier work this paper cites.

C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” J. Mach. Learn. Res. , vol. 21, no. 1, Jan. 2020

2020

Earlier work this paper cites.

T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y. Jernite, J. Plu, C. Xu, T. Le Scao, S. Gugger, M. Drame, Q. Lhoest, and A. Rush, “Transformers: State-of-the-art natural language processing,” in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , Q. Liu and D. Schlangen, Eds. Online: Association for Computational Linguistics, Oct. 2020, pp. 38–45. [Online]. Available: https://aclanthology.org/2020.emnlp-demos.6/

2020

Earlier work this paper cites.

P. Hase, S. Zhang, H. Xie, and M. Bansal, “Leakage-adjusted simulatability: Can models generate non-trivial explanations of their behavior in natural language?” in Findings of the Association for Computational Linguistics: EMNLP 2020 , T. Cohn, Y. He, and Y. Liu, Eds. Online: Association for Computational Linguistics, Nov. 2020, pp. 4351–4367. [Online]. Available: https://aclanthology.org/2020.findings-emnlp.390/

2020

Earlier work this paper cites.

A. Holzinger, A. Carrington, and H. Müller, “Measuring the Quality of Explanations: The System Causability Scale (SCS): Comparing Human and Machine Explanations,” KI - Künstliche Intelligenz , vol. 34, no. 2, pp. 193–198, Jun. 2020. [Online]. Available: http://link.springer.com/10.1007/s13218-020-00636-z

2020

Earlier work this paper cites.

T. Le Scao and A. Rush, “How many data points is a prompt worth?” in Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies . Online: Association for Computational Linguistics, 2021, pp. 2627–2636. [Online]. Available: https://aclanthology.org/2021.naacl-main.208

2021

Earlier work this paper cites.

W. Jin, Y. Cheng, Y. Shen, W. Chen, and X. Ren, “A good prompt is worth millions of parameters? low-resource prompt-based learning for vision-language models,” CoRR , vol. abs/2110.08484, 2021. [Online]. Available: https://arxiv.org/abs/2110.08484

Original

2021

Cited alongside, same era.

C. Wu, F. Wu, and Y. Huang, “One teacher is enough? pre-trained language model distillation from multiple teachers,” in Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 , C. Zong, F. Xia, W. Li, and R. Navigli, Eds. Online: Association for Computational Linguistics, Aug. 2021, pp. 4408–4413. [Online]. Available: https://aclanthology.org/2021.findings-acl.387/

2021

Cited alongside, same era.

R. He, L. Liu, H. Ye, Q. Tan, B. Ding, L. Cheng, J. Low, L. Bing, and L. Si, “On the effectiveness of adapter-based tuning for pretrained language model adaptation,” in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) . Association for Computational Linguistics, 2021. [Online]. Available: http://dx.doi.org/10.18653/v1/2021.acl-long.172

X. Wang, J. Wei, D. Schuurmans, Q. V. Le, E. H. Chi, S. Narang, A. Chowdhery, and D. Zhou, “Self-consistency improves chain of thought reasoning in language models,” in The Eleventh International Conference on Learning Representations , 2023. [Online]. Available: https://openreview.net/forum?id=1PL1NIMMrw

2023

Later among the works it cites.

H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. Canton Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, I. Kloumann, A. Korenev, P. Singh Koura, M.-A. Lachaux, T. Lavril, J. Lee, D. Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaylov, P. Mishra, I. Molybog, Y. Nie, A. Poulton, J. Reizenstein, R. Rungta, K. Saladi, A. Schelten, R. Silva, E. M. Smith, R. Subramanian, X. E. Tan, B. Tang, R. Taylor, A. Williams, J. X. Kuan, P. Xu, Z. Yan, I. Zarov, Y. Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, S. Edunov, and T. Scialom, “Llama 2: Open Foundation and Fine-Tuned Chat Models,” arXiv e-prints , p. arXiv:2307.09288, Jul. 2023

Original

2023

Later among the works it cites.

Prolific, “Prolific · Quickly find research participants you can trust.” 2023. [Online]. Available: https://www.prolific.com/

2023

Later among the works it cites.