2020

Learning from Rules Generalizing Labeled Exemplars

Awasthi, Abhijeet, Ghosh, Sabyasachi, Goyal, Rasna et al.

Understand

In many applications labeled data is not readily available, and needs to be collected via pain-staking human supervision.

We propose a rule-exemplar method for collecting human supervision to combine the efficiency of rules with the quality of instance labels.
The supervision is coupled such that it is both natural for humans and synergistic for learning.
We propose a training algorithm that jointly denoises rules via latent coverage variables, and trains the model through a soft implication loss over the coverage and label variables.

Built on

Maximum likelihood from incomplete data via the em algorithm
Arthur P Dempster, Nan M Laird, and Donald B Rubin · 1977
Earlier work this paper cites.
Fastus: A finite-state processor for information extraction from real-world text
Douglas E. Appelt, Jerry R. Hobbs, John Bear, David J. Israel, and Mabry Tyson · 1993
Earlier work this paper cites.
Combining labeled and unlabeled data with co-training
Avrim Blum and Tom Mitchell · 1998
Earlier work this paper cites.
Generating accurate rule sets without global optimization
Eibe Frank and Ian H. Witten · 1998
Earlier work this paper cites.
Gate: A framework and graphical development environment for robust nlp tools and applications
Hamish Cunningham · 2002
Earlier work this paper cites.
Learning question classifiers
Xin Li and Dan Roth · 2002
Earlier work this paper cites.
Trading convexity for scalability
R. Collobert, F. Sinz, J. Weston, and L. Bottou · 2006
Earlier work this paper cites.
Posterior regularization for structured latent variable models
Kuzman Ganchev, Jennifer Gillenwater, Ben Taskar, et al · 2010
Earlier work this paper cites.
Contributions to the study of sms spam filtering: new collection and results
Tiago A Almeida, José María G Hidalgo, and Akebo Yamakami · 2011
Earlier work this paper cites.
Query understanding enhanced by hierarchical parsing structures
Jingjing Liu, Panupong Pasupat, Yining Wang, Scott Cyphers, and Jim Glass · 2013
Earlier work this paper cites.
Tubespam: Comment spam filtering on youtube
Túlio C Alberto, Johannes V Lochter, and Tiago A Almeida · 2015
Earlier work this paper cites.

Similar

Training deep neural-networks using a noise adaptation layer
Jacob Goldberger and Ehud Ben-Reuven · 2016
Cited alongside, same era.
Harnessing deep neural networks with logic rules
Zhiting Hu, Xuezhe Ma, Zhengzhong Liu, Eduard Hovy, and Eric Xing · 2016
Cited alongside, same era.
Data programming: Creating large training sets, quickly
Alexander J Ratner, Christopher M De Sa, Sen Wu, Daniel Selsam, and Christopher Ré · 2016
Cited alongside, same era.
Robust loss functions under label noise for deep neural networks
Aritra Ghosh, Himanshu Kumar, and PS Sastry · 2017
Cited alongside, same era.
Revisiting unreasonable effectiveness of data in deep learning era
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta · 2017
Cited alongside, same era.
Learning from noisy large-scale datasets with minimal supervision
Andreas Veit, Neil Alldrin, Gal Chechik, Ivan Krasin, Abhinav Gupta, and Serge Belongie · 2017
Cited alongside, same era.
Bert: Pre-training of deep bidirectional transformers for language understanding
Original
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova · 2018
Cited alongside, same era.
Using rule-based labels for weak supervised learning: A chemnet for transferable chemical property prediction
Garrett B. Goh, Charles Siegel, Abhinav Vishnu, and Nathan Hodas · 2018
Cited alongside, same era.
Adventure: Adversarial training for textual entailment with knowledge-guided examples
Dongyeop Kang, Tushar Khot, Ashish Sabharwal, and Eduard Hovy · 2018
Cited alongside, same era.
Learning from noisy singly-labeled data
Ashish Khetan, Zachary C. Lipton, and Anima Anandkumar · 2018
Cited alongside, same era.
Deepproblog: Neural probabilistic logic programming
Robin Manhaeve, Sebastijan Dumancic, Angelika Kimmig, Thomas Demeester, and Luc De Raedt · 2018
Cited alongside, same era.
Masking: A new perspective of noisy supervision
Bo Han, Jiangchao Yao, Gang Niu, Mingyuan Zhou, Ivor Tsang, Ya Zhang, and Masashi Sugiyama
Cited in the paper.

Then

Adversarial data programming: Using gans to relax the bottleneck of curated labeled data
Arghya Pal and Vineeth N. Balasubramanian · 2018
Later among the works it cites.
Deep contextualized word representations
Original
Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer · 2018
Later among the works it cites.
Semi-supervised learning with declaratively specified entropy constraints
Haitian Sun, William W Cohen, and Lidong Bing · 2018
Later among the works it cites.
Joint optimization framework for learning with noisy labels
Daiki Tanaka, Daiki Ikami, Toshihiko Yamasaki, and Kiyoharu Aizawa · 2018
Later among the works it cites.
A semantic loss function for deep learning with symbolic knowledge
Jingyi Xu, Zilu Zhang, Tal Friedman, Yitao Liang, and Guy Van den Broeck · 2018
Later among the works it cites.
Generalized cross entropy loss for training deep neural networks with noisy labels
Zhilu Zhang and Mert Sabuncu · 2018
Later among the works it cites.
Snorkel drybell: A case study in deploying weak supervision at industrial scale
Stephen H. Bach, Daniel Rodriguez, Yintao Liu, Chong Luo, Haidong Shao, Cassandra Xia, Souvik Sen, Alexander Ratner, Braden Hancock, Houman Alborzi, Rahul Kuchhal, Christopher Ré, and Rob Malkin · 2019
Later among the works it cites.
UCI machine learning repository, 2019
Dheeru Dua and Casey Graff · 2019
Later among the works it cites.
DL2: Training and querying neural networks with logic
Marc Fischer, Mislav Balunovic, Dana Drachsler-Cohen, Timon Gehr, Ce Zhang, and Martin Vechev · 2019
Later among the works it cites.
Learning with bad training data via iterative trimmed loss minimization
Yanyao Shen and Sujay Sanghavi · 2019
Later among the works it cites.
The language of generalization
Michael Henry Tessler and Noah D. Goodman · 2019
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…