2021

PnPOOD : Out-Of-Distribution Detection for Text Classification via Plug andPlay Data Augmentation

Rawat, Mrinal, Hebbalaguppe, Ramya, Vig, Lovekesh

Understand

While Out-of-distribution (OOD) detection has been well explored in computer vision, there have been relatively few prior attempts in OOD detection for NLP classification.

In this paper we argue that these prior attempts do not fully address the OOD problem and may suffer from data leakage and poor calibration of the resulting models.
We present PnPOOD, a data augmentation technique to perform OOD detection via out-of-domain sample generation using the recently proposed Plug and Play Language Model (Dathathri et al., 2020).
Our method generates high quality discriminative samples close to the class boundaries, resulting in accurate OOD detection at test time.

Built on

An evaluation dataset for intent classification and out-of-scope prediction
Original
Larson, S., Mahendran, A., Peper, J. J., Clarke, C., Lee, A., Hill, P., Kummerfeld, J. K., Leach, K., Laurenzano, M. A., Tang, L., and Mars, J · 1909
Earlier work this paper cites.
Newsweeder: Learning to filter netnews
Lang, K · 1995
Earlier work this paper cites.
Crafting papers on machine learning
Langley, P · 2000
Earlier work this paper cites.
Simple and principled uncertainty estimation with deterministic deep learning via distance awareness
Original
Liu, J. Z., Lin, Z., Padhy, S., Tran, D., Bedrax-Weiss, T., and Lakshminarayanan, B · 2006
Earlier work this paper cites.
Svm-knn: Discriminative nearest neighbor classification for visual category recognition
Zhang, H., Berg, A. C., Maire, M., and Malik, J · 2006
Earlier work this paper cites.
Recursive deep models for semantic compositionality over a sentiment treebank
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., and Potts, C · 2013
Earlier work this paper cites.
Complexity of Word Collocation Networks: A Preliminary Structural Analysis
Lahiri, S · 2014
Earlier work this paper cites.
A large annotated corpus for learning natural language inference
Bowman, S. R., Angeli, G., Potts, C., and Manning, C. D · 2015
Earlier work this paper cites.
Distilling the knowledge in a neural network
Original
Hinton, G., Vinyals, O., and Dean, J · 2015
Earlier work this paper cites.

Similar

Multi30k: Multilingual english-german image descriptions
Elliott, D., Frank, S., Sima’an, K., and Specia, L · 2016
Cited alongside, same era.
Dropout as a bayesian approximation: Representing model uncertainty in deep learning
Gal, Y. and Ghahramani, Z · 2016
Cited alongside, same era.
On calibration of modern neural networks
Guo, C., Pleiss, G., Sun, Y., and Weinberger, K. Q · 2017
Cited alongside, same era.
A baseline for detecting misclassified and out-of-distribution examples in neural networks
Hendrycks, D. and Gimpel, K · 2017
Cited alongside, same era.
Simple and scalable predictive uncertainty estimation using deep ensembles
Lakshminarayanan, B., Pritzel, A., and Blundell, C · 2017
Cited alongside, same era.
Training confidence-calibrated classifiers for detecting out-of-distribution samples
Original
Lee, K., Lee, H., Lee, K., and Shin, J · 2017
Cited alongside, same era.
Enhancing the reliability of out-of-distribution image detection in neural networks
Original
Liang, S., Li, Y., and Srikant, R · 2017
Cited alongside, same era.
BERT: pre-training of deep bidirectional transformers for language understanding
Original
Devlin, J., Chang, M., Lee, K., and Toutanova, K · 2018
Cited alongside, same era.
A simple unified framework for detecting out-of-distribution samples and adversarial attacks
Lee, K., Lee, K., Lee, H., and Shin, J · 2018
Cited alongside, same era.

Then

Why relu networks yield high-confidence predictions far away from the training data and how to mitigate the problem
Hein, M., Andriushchenko, M., and Bitterwolf, J · 2019
Later among the works it cites.
Benchmarking neural network robustness to common corruptions and perturbations
Hendrycks, D. and Dietterich, T · 2019
Later among the works it cites.
Deep anomaly detection with outlier exposure
Hendrycks, D., Mazeika, M., and Dietterich, T · 2019
Later among the works it cites.
Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with dirichlet calibration
Original
Kull, M., Perello-Nieto, M., Kängsepp, M., Song, H., Flach, P., et al · 2019
Later among the works it cites.
Language models are unsupervised multitask learners
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I · 2019
Later among the works it cites.
Likelihood ratios for out-of-distribution detection
Ren, J., Liu, P. J., Fertig, E., Snoek, J., Poplin, R., Depristo, M., Dillon, J., and Lakshminarayanan, B · 2019
Later among the works it cites.
Plug and play language models: A simple approach to controlled text generation
Original
Dathathri, S., Madotto, A., Lan, J., Hung, J., Frank, E., Molino, P., Yosinski, J., and Liu, R · 2020
Later among the works it cites.
Transformers: State-of-the-art natural language processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., Davison, J., Shleifer, S., von Platen, P., Ma, C., Jernite, Y., Plu, J., Xu, C., Scao, T. L., Gugger, S., Drame, M., Lhoest, Q., and Rush, A. M · 2020
Later among the works it cites.
On-manifold adversarial data augmentation improves uncertainty calibration
Patel, K., Beluch, W., Zhang, D., Pfeiffer, M., and Yang, B · 2021
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…