2020

A Survey of Active Learning for Text Classification using Deep Neural Networks

Schröder, Christopher, Niekler, Andreas

Understand

Natural language processing (NLP) and neural networks (NNs) have both undergone significant changes in recent years.

For active learning (AL) purposes, NNs are, however, less commonly used -- despite their current popularity.
By using the superior text classification performance of NNs for AL, we can either increase a model's performance using the same amount of data or reduce the data and therefore the required annotation efforts while keeping the same performance.
We review AL for text classification using deep neural networks (DNNs) and elaborate on two main causes which used to hinder the adoption: (a) the inability of NNs to provide reliable uncertainty estimates, on which the most commonly used query strategies rely, and (b) the challenge of training DNNs on small data.

Built on

“Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Bounds” arXiv: 1906.03671
Original
Jordan. Ash et al · 1906
Earlier work this paper cites.
“Discriminative Active Learning” arXiv: 1907.06347
Original
Daniel Gissin and Shai Shalev-Shwartz · 1907
Earlier work this paper cites.
“RoBERTa: A Robustly Optimized BERT Pretraining Approach” arXiv: 1907.11692
Original
Yinhan Liu et al · 1907
Earlier work this paper cites.
“Dropout: A Simple Way to Prevent Neural Networks from Overfitting”
Nitish Srivastava et al · 1958
Earlier work this paper cites.
“Direct Transfer of Learned Information Among Neural Networks.”
Lorien. Pratt, Jack Mostow, Candace. Kamm and Ace. Kamm · 1991
Earlier work this paper cites.
“The Evidence Framework Applied to Classification Networks”
David MacKay · 1992
Earlier work this paper cites.
“Towards Language Independent Automated Learning of Text Categorization Models”
Chidanand Apté, Fred Damerau and Sholom. Weiss · 1994
Earlier work this paper cites.
“Improving Generalization with Active Learning”
David Cohn, Les Atlas and Richard Ladner · 1994
Earlier work this paper cites.
“Neural Network Ensembles, Cross Validation and Active Learning”
Anders Krogh and Jesper Vedelsby · 1994
Earlier work this paper cites.
“A Sequential Algorithm for Training Text Classifiers”
David. Lewis and William. Gale · 1994
Earlier work this paper cites.
“Learning Many Related Tasks at the Same Time with Backpropagation”
Rich Caruana · 1995
Earlier work this paper cites.
“Learning Local Error Bars for Nonlinear Regression”
David. Nix and Andreas. Weigend · 1995
Earlier work this paper cites.
“Practical Confidence and Prediction Intervals”
Tom Heskes · 1996
Earlier work this paper cites.
“Long Short-Term Memory”
Sepp Hochreiter and Jürgen Schmidhuber · 1997
Earlier work this paper cites.
“A Probabilistic Analysis of the Rocchio Algorithm with TFIDF for Text Categorization”
Thorsten Joachims · 1997
Earlier work this paper cites.
“Reuters-21578 corpus” Online. Visited on 02/14/2020, http://www.daviddlewis.com/resources/testcollections/reuters21578/ , 1997
David. Lewis · 1997
Earlier work this paper cites.
“Active Learning with Committees for Text Categorization”
Ray Liere and Prasad Tadepalli · 1997
Earlier work this paper cites.
“Confidence and prediction intervals for neural network ensembles”
John Carney, Padraig Cunningham and Umesh Bhagwan · 1999
Earlier work this paper cites.
“A patent search and classification system”
Leah. Larkey · 1999
Earlier work this paper cites.
“Text Classification from Labeled and Unlabeled Documents using EM”
Kamal Nigam, Andrew McCallum, Sebastian Thrun and Tom. Mitchell · 2000
Earlier work this paper cites.
“Less is More: Active Learning with Support Vector Machines”
Greg Schohn and David Cohn · 2000
Earlier work this paper cites.
“Toward Optimal Active Learning through Sampling Estimation of Error Reduction”
Nicholas Roy and Andrew McCallum · 2001
Earlier work this paper cites.
“Support Vector Machine Active Learning with Applications to Text Classification”
Simon Tong and Daphne Koller · 2001
Earlier work this paper cites.
“Learning Question Classifiers”
Xin Li and Dan Roth · 2002
Earlier work this paper cites.
“Automated Categorization in the International Patent Classification”
C.. Fall, A. Törcsvári, K. Benzineb and G. Karetka · 2003
Earlier work this paper cites.
“Representative sampling for text classification using support vector machines”
Zhao Xu et al · 2003
Earlier work this paper cites.
“Mining and Summarizing Customer Reviews”
Minqing Hu and Bing Liu · 2004
Earlier work this paper cites.
“RCV1: A New Benchmark Collection for Text Categorization Research”
David. Lewis, Yiming Yang, Tony. Rose and Fan Li · 2004
Earlier work this paper cites.
“Diverse Ensembles for Active Learning”
Prem Melville and Raymond. Mooney · 2004
Earlier work this paper cites.
“Active Learning Using Pre-Clustering”
Hieu. Nguyen and Arnold Smeulders · 2004
Earlier work this paper cites.
“A Sentimental Education: Sentiment Analysis Using Subjectivity Summarization Based on Minimum Cuts”
Bo Pang and Lillian Lee · 2004
Earlier work this paper cites.
“Multi-Criteria-Based Active Learning for Named Entity Recognition”
Dan Shen et al · 2004
Earlier work this paper cites.
“A case-based technique for tracking concept drift in spam filtering”
Sarah Delany, Pádraig Cunningham, Alexey Tsymbal and Lorcan Coyle · 2005
Earlier work this paper cites.
“Framewise phoneme classification with bidirectional LSTM and other neural network architectures”
Alex Graves and Jürgen Schmidhuber · 2005
Earlier work this paper cites.
“AG’s Corpus of News Articles” Online; visited on 02/11/2020, http://groups.di.unipi.it/˜gulli/AG_corpus_of_news_articles.html , 2005
Antonio Gulli · 2005
Earlier work this paper cites.
“Seeing Stars: Exploiting Class Relationships for Sentiment Categorization with Respect to Rating Scales”
Bo Pang and Lillian Lee · 2005
Earlier work this paper cites.

Similar

“Reducing the dimensionality of data with neural networks”
Geoffrey Hinton and Ruslan Salakhutdinov · 2006
Cited alongside, same era.
“Large-Scale Text Categorization by Batch Mode Active Learning”
Steven.. Hoi, Rong Jin and Michael. Lyu · 2006
Cited alongside, same era.
“K-Means++: The Advantages of Careful Seeding”
David Arthur and Sergei Vassilvitskii · 2007
Cited alongside, same era.
“Biographies, Bollywood, Boom-boxes and Blenders: Domain Adaptation for Sentiment Classification”
John Blitzer, Mark Dredze and Fernando Pereira · 2007
Cited alongside, same era.
“Discriminative Batch Mode Active Learning”
Yuhong Guo and Dale Schuurmans · 2007
Cited alongside, same era.
“A shared task involving multi-label classification of clinical free text”
John. Pestian et al · 2007
Cited alongside, same era.
“Multiple-Instance Active Learning”
Burr Settles, Mark Craven and Soumya Ray · 2007
Cited alongside, same era.
“Hierarchical Sampling for Active Learning”
Sanjoy Dasgupta and Daniel Hsu · 2008
Cited alongside, same era.
“A Holistic Lexicon-Based Approach to Opinion Mining”
Xiaowen Ding, Bing Liu and Philip. Yu · 2008
Cited alongside, same era.
“Automatic Online News Issue Construction in Web Environment”
Canhui Wang, Min Zhang, Shaoping Ma and Liyun Ru · 2008
Cited alongside, same era.
“Active Learning for Multilingual Statistical Machine Translation”
Gholamreza Haffari and Anoop Sarkar · 2009
Cited alongside, same era.
“A literature survey of active machine learning in the context of natural language processing”, SICS Technical Report
Fredrik Olsson · 2009
Cited alongside, same era.

Then

“Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning”
Yarin Gal and Zoubin Ghahramani · 2016
Later among the works it cites.
“Active Learning for Text Classification with Reusability”
Rong Hu, Brian Mac and Sarah Delany · 2016
Later among the works it cites.
“FastText.zip: Compressing text classification models” arXiv: 1612.03651
Original
Armand Joulin et al · 2016
Later among the works it cites.
“ALTO: Active Learning with Topic Overviews for Speeding Label Induction and Document Labeling”
Forough Poursabzi-Sangdeh, Jordan Boyd-Graber, Leah Findlater and Kevin Seppi · 2016
Later among the works it cites.
“Learning Algorithms for Active Learning”
Philip Bachman, Alessandro Sordoni and Adam Trischler · 2017
Later among the works it cites.
“On Calibration of Modern Neural Networks”
Chuan Guo, Geoff Pleiss, Yu Sun and Kilian. Weinberger · 2017
Later among the works it cites.
“Bag of Tricks for Efficient Text Classification”
Armand Joulin, Edouard Grave, Piotr Bojanowski and Tomas Mikolov · 2017
Later among the works it cites.
“Learning Active Learning from Data”
Ksenia Konyushkova, Sznitman Raphael and Pascal Fua · 2017
Later among the works it cites.
“Simple and Scalable Predictive Uncertainty Estimation Using Deep Ensembles”
Balaji Lakshminarayanan, Alexander Pritzel and Charles Blundell · 2017
Later among the works it cites.
“Regularizing and optimizing LSTM language models”
Original
Stephen Merity, Nitish Keskar and Richard Socher · 2017
Later among the works it cites.
“Evidence-Based Uncertainty Sampling for Active Learning”
Manali Sharma and Mustafa Bilgic · 2017
Later among the works it cites.
“The Effectiveness of Data Augmentation in Image Classification using Deep Learning”
Original
Jason Wang and Luis Perez · 2017
Later among the works it cites.
“Active Discriminative Text Representation Learning”
Ye Zhang, Matthew Lease and Byron. Wallace · 2017
Later among the works it cites.
“Adversarial Feature Matching for Text Generation”
Yizhe Zhang et al · 2017
Later among the works it cites.
“Generative Adversarial Active Learning”
Original
Jia-Jie Zhu and José Bento · 2017
Later among the works it cites.
“Deep Active Learning for Text Classification”
Bang An, Wenjun Wu and Huimin Han · 2018
Later among the works it cites.
“Stability of Topic Modeling via Matrix Factorization”
Mark Belford, Brian Mac and Derek Greene · 2018
Later among the works it cites.
“Support Vector Machine Active Learning Algorithms with Query-by-Committee Versus Closest-to-Hyperplane Selection”
Michael Bloodgood · 2018
Later among the works it cites.
“Adversarial Active Learning for Deep Networks: a Margin Based Approach”
Original
Melanie Ducoffe and Frederic Precioso · 2018
Later among the works it cites.
“Universal Language Model Fine-tuning for Text Classification”
Jeremy Howard and Sebastian Ruder · 2018
Later among the works it cites.
“Deep contextualized word representations” arXiv: 1802.05365
Original
Matthew. Peters et al · 2018
Later among the works it cites.
“Active Learning for Convolutional Neural Networks: A Core-Set Approach”
Ozan Sener and Silvio Savarese · 2018
Later among the works it cites.
“Deep Active Learning for Named Entity Recognition”
Original
Yanyao Shen et al · 2018
Later among the works it cites.
“Deep Bayesian Active Learning for Natural Language Processing: Results of a Large-Scale Empirical Study”
Aditya Siddhant and Zachary. Lipton · 2018
Later among the works it cites.
“Improving Deep Learning with Generic Data Augmentation”
Luke Taylor and Geoff Nitschke · 2018
Later among the works it cites.
“Transformer-XL: Attentive Language Models beyond a Fixed-Length Context”
Zihang Dai et al · 2019
Later among the works it cites.
“BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding”
Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova · 2019
Later among the works it cites.
“Practical Obstacles to Deploying Active Learning”
David Lowell, Zachary. Lipton and Byron. Wallace · 2019
Later among the works it cites.
“Sampling Bias in Deep Active Classification: An Empirical Study”
Ameya Prabhu, Charles Dognin and Maneesh Singh · 2019
Later among the works it cites.
“Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks”
Nils Reimers and Iryna Gurevych · 2019
Later among the works it cites.
“XLNet: Generalized Autoregressive Pretraining for Language Understanding”
Zhilin Yang et al · 2019
Later among the works it cites.
“ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators”
Original
Kevin Clark, Minh-Thang Luong and Quoc. Le · 2020
Closest in time.
“Investigating the Effectiveness of Representations Based on Pretrained Transformer-based Language Models in Active Learning for Labelling Text Datasets”
Original
Jinghui Lu and Brian MacNamee · 2020
Closest in time.
“DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter”
Original
Victor Sanh, Lysandre Debut, Julien Chaumond and Thomas Wolf · 2020
Closest in time.
“How to Fine-Tune BERT for Text Classification?”
Original
Chi Sun, Xipeng Qiu, Yige Xu and Xuanjing Huang · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…