Hands-On Tutorial: Labeling with LLM and Human-in-the-Loop

Large language models in medicine

Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting. 2023 · 1940

Earlier work this paper cites.

Rating the ratings: Assessing the psychometric quality of rating data

Frank E Saal, Ronald G Downey, and Mary A Lahey. 1980 · 1980

Earlier work this paper cites.

A sequential algorithm for training text classifiers

David D. Lewis and William A. Gale. 1994 · 1994

Earlier work this paper cites.

Scaling laws for neural language models

Original

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020 · 2001

Earlier work this paper cites.

Active learning to recognize multiple types of plankton

Tong Luo, Kurt Kramer, Dmitry B. Goldgof, Lawrence O. Hall, Scott Samson, Andrew Remsen, and Thomas Hopkins. 2005 · 2005

Earlier work this paper cites.

Uncertainty-aware self-training for text classification with few labels

Original

Subhabrata Mukherjee and Ahmed Hassan Awadallah. 2020 · 2006

Earlier work this paper cites.

A survey of active learning for text classification using deep neural networks

Original

Christopher Schröder and Andreas Niekler. 2020 · 2008

Earlier work this paper cites.

Best practices for managing data annotation projects

Original

Tina Tseng, Amanda Stent, and Domenic Maida. 2020 · 2009

Earlier work this paper cites.

Corpus annotation through crowdsourcing: Towards best practice guidelines

Marta Sabou, Kalina Bontcheva, Leon Derczynski, and Arno Scharl. 2014 · 2014

Earlier work this paper cites.

Introduction: The Handbook of Linguistic Annotation

Nancy Ide and James Pustejovsky. 2017 · 2017

Earlier work this paper cites.

Deep batch active learning by diverse, uncertain gradient lower bounds

Jordan T. Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agarwal. 2020 · 2020

Earlier work this paper cites.

Unsupervised cross-lingual representation learning at scale

Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Édouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020 · 2020

Earlier work this paper cites.

Active learning for bert: an empirical study

Liat Ein Dor, Alon Halfon, Ariel Gera, Eyal Shnarch, Lena Dankin, Leshem Choshen, Marina Danilevsky, Ranit Aharonov, Yoav Katz, and Noam Slonim. 2020 · 2020

Earlier work this paper cites.

Crowdsourcing practice for efficient data labeling: Aggregation, incremental relabeling, and pricing

Alexey Drutsa, Valentina Fedorova, Dmitry Ustalov, Olga Megorskaya, Evfrosiniya Zerminova, and Daria Baidakova. 2020 · 2020

Earlier work this paper cites.

Structured pruning of large language models

Ziheng Wang, Jeremy Wohlwend, and Tao Lei. 2020 · 2020

Earlier work this paper cites.

Crowdsourcing natural language data at scale: A hands-on tutorial

Alexey Drutsa, Dmitry Ustalov, Valentina Fedorova, Olga Megorskaya, and Daria Baidakova. 2021 · 2021

Cited alongside, same era.

Active learning by acquiring contrastive examples

Katerina Margatina, Giorgos Vernikos, Loïc Barrault, and Nikolaos Aletras. 2021 · 2021

Cited alongside, same era.

Crowdsourcing beyond annotation: Case studies in benchmark data collection

Alane Suhr, Clara Vania, Nikita Nangia, Maarten Sap, Mark Yatskar, Samuel R. Bowman, and Yoav Artzi. 2021 · 2021

Cited alongside, same era.

Putting humans in the natural language processing loop: A survey

Zijie J Wang, Dongjin Choi, Shenyu Xu, and Diyi Yang. 2021 · 2021

Cited alongside, same era.

Jury learning: Integrating dissenting voices into machine learning models

Mitchell L Gordon, Michelle S Lam, Joon Sung Park, Kayur Patel, Jeff Hancock, Tatsunori Hashimoto, and Michael S Bernstein. 2022 · 2022

Cited alongside, same era.

The curse of recursion: Training on generated data makes models forget

Original

Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Yarin Gal, Nicolas Papernot, and Ross Anderson. 2023 · 2023

Later among the works it cites.

To aggregate or not? learning with separate noisy labels

Jiaheng Wei, Zhaowei Zhu, Tianyi Luo, Ehsan Amid, Abhishek Kumar, and Yang Liu. 2023 · 2023

Later among the works it cites.

LLMaAA: Making Large Language Models as Active Annotators

Ruoyu Zhang, Yanzeng Li, Yongliang Ma, Ming Zhou, and Lei Zou. 2023 · 2023

Later among the works it cites.

Bias and fairness in large language models: A survey

Isabel O Gallegos, Ryan A Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen K Ahmed. 2024 · 2024

Closest in time.

Analyzing dataset annotation quality management in the wild

Jan-Christoph Klie, Richard Eckart de Castilho, and Iryna Gurevych. 2024 · 2024

Closest in time.