2022

Self-Training: A Survey

Amini, Massih-Reza, Feofanov, Vasilii, Pauletto, Loic et al.

Understand

Semi-supervised algorithms aim to learn prediction functions from a small set of labeled observations and a large set of unlabeled observations.

Because this framework is relevant in many applications, they have received a lot of interest in both academia and industry.
Among the existing techniques, self-training methods have undoubtedly attracted greater attention in recent years.
These models are designed to find the decision boundary on low density regions without making additional assumptions about the data distribution, and use the unsigned output score of a learned classifier, or its margin, as an indicator of confidence.

Built on

Adaptive communication receivers
H. Scudder · 1965
Earlier work this paper cites.
A theory of the learnable
L.G. Valiant · 1984
Earlier work this paper cites.
On the exponential value of labeled samples
V. Castelli and T.M. Cover · 1995
Earlier work this paper cites.
Boosting the margin: A new explanation for the effectiveness of voting methods
R.E. Schapire, Y. Freund, P. Barlett, and W. S. Lee · 1997
Earlier work this paper cites.
Combining labeled and unlabeled data with co-training
A. Blum and T. Mitchell · 1998
Earlier work this paper cites.
Statistical Learning Theory
V. Vapnik · 1998
Earlier work this paper cites.
Transductive inference for text classification using support vector machines
T. Joachims · 1999
Earlier work this paper cites.
Analyzing the effectiveness and applicability of co-training
K. Nigam and R. Ghani · 2000
Earlier work this paper cites.
Semi-supervised logistic regression
M.-R. Amini and P. Gallinari · 2002
Earlier work this paper cites.
Semi-supervised learning using gaussian fields and harmonic functions
X. Zhu, Z. Ghahramani, and J.D. Lafferty · 2003
Earlier work this paper cites.
Semi-supervised learning on riemannian manifolds
M. Belkin and P. Niyogi · 2004
Earlier work this paper cites.
Combining active and semi-supervised learning for spoken language understanding
G. Tür, D. Tür, and R.-E. Schapire · 2005
Earlier work this paper cites.
An augmented PAC model for semi-supervised learning
M.-F. Balcan and A. Blum · 2006
Earlier work this paper cites.
A transductive bound for the voted classifier with an application to semi-supervised learning
M.-R. Amini, F. Laviolette, and N. Usunier · 2008
Earlier work this paper cites.
Does unlabeled data provably help? worst-case analysis of the sample complexity of semi-supervised learning
S. Ben-David, T. Lu, and D. Pál · 2008
Earlier work this paper cites.
Semi-supervised document classification with a mislabeling error model
A. Krithara, M.-R. Amini, J.-M. Renders, and C. Goutte · 2008
Earlier work this paper cites.
Unlabeled data: Now it helps, now it doesn’t
A. Singh, R. Nowak, and J. Zhu · 2008
Earlier work this paper cites.
Semi-Supervised Learning
O. Chapelle, B. Schölkopf, and A. Zien · 2010
Earlier work this paper cites.
Scikit-learn: Machine learning in python
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, and V. Dubourg · 2011
Earlier work this paper cites.
Pseudo-label : The simple and efficient semi-supervised learning method for deep neural networks
D.-H. Lee · 2013
Earlier work this paper cites.
Manifold regularization and semi-supervised learning: Some theoretical analyses
P. Niyogi · 2013
Earlier work this paper cites.
Semi-supervised learning with deep generative models
D.P. Kingma, S. Mohamed, D. Jimenez, and M. Welling · 2014
Earlier work this paper cites.
Learning with Partially Labeled and Interdependent Data
M.-R. Amini and N. Usunier · 2015
Earlier work this paper cites.
Domain-adversarial training of neural networks
Y. Ganin, E. Ustinova, H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, M. March, and V. Lempitsky · 2016
Earlier work this paper cites.
Seqping: gene prediction pipeline for plant genomes using self-training gene models and transcriptomic data
K.-L. Chan, R. Rosli, T. V. Tatarinova, M. Hogan, M. Firdaus-Raih, and L. Eng-Ti Low · 2017
Earlier work this paper cites.
Aggressive sampling for multi-class to binary reduction with applications to text classification
B. Joshi, M.-R. Amini, I. Partalas, F. Iutzeler, and Y. Maximov · 2017
Earlier work this paper cites.
Temporal ensembling for semi-supervised learning
S. Laine and T. Aila · 2017
Earlier work this paper cites.
Making deep neural networks robust to label noise: A loss correction approach
G. Patrini, A. Rozza, A. K. Menon, R. Nock, and L. Qu · 2017
Earlier work this paper cites.
Asymmetric tri-training for unsupervised domain adaptation
K. Saito, Y. Ushiku, and T. Harada · 2017
Earlier work this paper cites.
Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results
A. Tarvainen and H. Valpola · 2017
Earlier work this paper cites.
Recovery guarantees for one-hidden-layer neural networks
K. Zhong, Z. Song, P. Jain, P. L. Bartlett, and I. S. Dhillon · 2017
Earlier work this paper cites.
Co-teaching: Robust training of deep neural networks with extremely noisy labels
B. Han, Q. Yao, X. Yu, G. Niu, M. Xu, W. Hu, I. Tsang, and M. Sugiyama · 2018
Earlier work this paper cites.
Rademacher complexity bounds for a penalized multi-class semi-supervised algorithm
Y. Maximov, M.-R. Amini, and Z. Harchaoui · 2018
Earlier work this paper cites.
Transductive semi-supervised deep learning using min-max features
W. Shi, Y. Gong, C. Ding, Z. Ma, X. Tao, and N. Zheng · 2018
Earlier work this paper cites.

Similar

Unsupervised domain adaptation for semantic segmentation via class-balanced self-training
Y. Zou, Z. Yu, B. Kumar, and J. Wang · 2018
Cited alongside, same era.
Computational translation of genomic responses from experimental model systems to humans
D. K. Brubaker, E. A Proctor, K. M. Haigis, and D.A. Lauffenburger · 2019
Cited alongside, same era.
Transductive bounds for the multi-class majority vote classifier
V. Feofanov, E. Devijver, and M.-R. Amini · 2019
Cited alongside, same era.
Virtual adversarial training: A regularization method for supervised and semi-supervised learning
T. Miyato, S. Maeda, M. Koyama, and S. Ishii · 2019
Cited alongside, same era.
Self-training avoids using spurious features under domain shift
Y. Chen, C. Wei, A. Kumar, and T. Ma · 2020
Cited alongside, same era.
Self-training for end-to-end speech recognition
J. Kahn, A. Lee, and A. Hannun · 2020
Cited alongside, same era.
Understanding self-training for gradual domain adaptation
A. Kumar, T. Ma, and P. Liang · 2020
Cited alongside, same era.
Uncertainty-aware self-training for few-shot text classification
S. Mukherjee and A. Awadallah · 2020
Cited alongside, same era.
Fixmatch: Simplifying semi-supervised learning with consistency and confidence
K. Sohn, D. Berthelot, N. Carlini, Z. Zhang, H. Zhang, C. Raffel, E. Cubuk, A. Kurakin, and C.-L. Li · 2020
Cited alongside, same era.
A survey on semi-supervised learning
J. E. Van Engelen and H. H. Hoos · 2020
Cited alongside, same era.
Single-/multi-source cross-lingual ner via teacher-student learning on unlabeled data in target language
Q. Wu, Z. Lin, B. F. Karlsson, J.-G. Lou, and B. Huang · 2020
Cited alongside, same era.
Self-training with noisy student improves imagenet classification
Q. Xie, M.-T. Luong, E. H. Hovy, and Q. V. Le · 2020
Cited alongside, same era.

Then

How unlabeled data improve generalization in self-training? a one-hidden-layer theoretical analysis
S. Zhang, M. Wang, S. Liu, P.-Y. Chen, and J. Xiong · 2022
Closest in time.
Making more of little data: Improving low-resource automatic speech recognition using data augmentation
M. Bartelds, N. San, B. McDonnell, D. Jurafsky, and M. Wieling · 2023
Closest in time.
Broadening BERT vocabulary for knowledge graph construction using wikipedia2vec
D. Biswas, S. Linzbach, D. Dimitrov, H. Jabeen, and S. Dietze · 2023
Closest in time.
Softmatch: Addressing the quantity-quality tradeoff in semi-supervised learning
H. Chen, R. Tao, Y. Fan, Y. Wang, J. Wang, B. Schiele, X. Xie, B. Raj, and M. Savvides · 2023
Closest in time.
Boxteacher: Exploring high-quality pseudo labels for weakly supervised instance segmentation
T. Cheng, X. Wang, S. Chen, Q. Zhang, and W. Liu · 2023
Closest in time.
Transductive learning for textual few-shot classification in API-based embedding models
P. Colombo, V. Pellegrain, M. Boudiaf, M. Tami, V. Storchan, I. Ayed, and P. Piantanida · 2023
Closest in time.
Corrupted image modeling for self-supervised visual pre-training
Y. Fang, L. Dong, H. Bao, X. Wang, and F. Wei · 2023
Closest in time.
Joint speech transcription and translation: Pseudo-labeling with out-of-distribution data
M. Gheini, T. Likhomanenko, M. Sperber, and H. Setiawan · 2023
Closest in time.
Reinforced self-training (rest) for language modeling, 2023
C. Gulcehre, T. Le Paine, S. Srinivasan, K. Konyushkova, L. Weerts, A. Sharma, A. Siddhant, A. Ahern, M. Wang, C. Gu, W. Macherey, A. Doucet, O. Firat, and N. de Freitas · 2023
Closest in time.
Self-training of halfspaces with generalization guarantees under massart mislabeling noise model
L. Hadjadj, M.-R. Amini, and S. Louhichi · 2023
Closest in time.
Semi-supervised bidirectional alignment for remote sensing cross-domain scene classification
W. Huang, Y. Shi, Z. Xiong, Q. Wang, and X. X. Zhu · 2023
Closest in time.
A self-training subspace clustering algorithm based on adaptive confidence for gene expression data
Dan Li, Hongnan Liang, Pan Qin, and Jia Wang · 2023
Closest in time.
Unbiased multiple instance learning for weakly supervised video anomaly detection
H. Lv, Z. Yue, Q. Sun, B. Luo, Z. Cui, and H. Zhang · 2023
Closest in time.
Know your self-supervised learning: A survey on image-based generative and discriminative training
U. Ozbulak, H. J. Lee, B. Boga, E. Timothy Anzaku, H. Park, A. Van Messem, W. De Neve, and J. Vankerschaver · 2023
Closest in time.
Faxmatch: Multi-curriculum pseudo-labeling for semi-supervised medical image classification
Z. Peng, D. Zhang, S. Tian, W. Wu, L. Yu, S. Zhou, and S. Huang · 2023
Closest in time.
Self-Supervised Anomaly Detection with Neural Transformations
C. Qiu · 2023
Closest in time.
Multiclass anomaly detection of bridge monitoring data with data migration between different bridges for balancing data
C. Qu, H. Zhang, R. Zhang, S. Zou, L. Huang, and H. Li · 2023
Closest in time.
Fgbcnn: A unified bilinear architecture for learning a fine-grained feature representation in facial expression recognition
N. Shabbir and R. Kumar Rout · 2023
Closest in time.
A novel self-training approach for low-resource speech recognition
S. Singh, F. Hou, and R. Wang · 2023
Closest in time.
Are labels informative in semi-supervised learning? Estimating and leveraging the missing-data mechanism
A. Sportisse, H. Schmutz, O. Humbert, C. Bouveyron, and P.-A. Mattei · 2023
Closest in time.
Dyannet: A scene dynamicity guided self-trained video anomaly detection network
K. Vijay Thakare, Y. Raghuwanshi, D. P. Dogra, H. Choi, and I.-J. Kim · 2023
Closest in time.
Freematch: Self-adaptive thresholding for semi-supervised learning
Y. Wang, H. Chen, Q. Heng, W. Hou, Y. Fan, Z. Wu, J. Wang, M. Savvides, T. Shinozaki, B. Raj, B. Schiele, and X. Xie · 2023
Closest in time.
A comprehensive survey of image augmentation techniques for deep learning
M. Xu, S. Yoon, A. Fuentes, and D. S. Park · 2023
Closest in time.
A survey on deep semi-supervised learning
X. Yang, Z. Song, I. King, and Z. Xu · 2023
Closest in time.
Topological identification and interpretation for single-cell gene regulation elucidation across multiple platforms using scMGCA
Z. Yu, Y. Su, Y. Lu, Y. Yang, F. Wang, S. Zhang, Y. Chang, K.-C. Wong, and X. Li · 2023
Closest in time.
Data-efficient active learning for structured prediction with partial annotation and self-training
Z. Zhang, E. Strubell, and E. H. Hovy · 2023
Closest in time.
Curriculum point prompting for weakly-supervised referring image segmentation
Qiyuan Dai and Sibei Yang · 2024
Closest in time.
Multi-class probabilistic bounds for majority vote classifiers with partially labeled data
V. Feofanov, E. Devijver, and M.-R. Amini · 2024
Closest in time.
Consistency-based semi-supervised learning for oriented object detection
R. Fu, C. Chen, S. Yan, X. Wang, and H. Chen · 2024
Closest in time.
Leveraging ensemble diversity for robust self-training in the presence of sample selection bias
A. Odonnat, V. Feofanov, and I. Redko · 2024
Closest in time.
Adversarial self-training improves robustness and generalization for gradual domain adaptation
L. Shi and W. Liu · 2024
Closest in time.
Beyond human data: Scaling self-training for problem-solving with language models
A. Singh, J. D Co-Reyes, R. Agarwal, A. Anand, P. Patil, X. Garcia, P. J. Liu, J. Harrison, J. Lee, K. Xu, A. T. Parisi, A. Kumar, A; A Alemi, A. Rizkowsky, A. Nova, B. Adlam, B. Bohnet, G.F. Elsayed, H. Sedghi, I. Mordatch, I.Simpson, I. Gur, J. Snoek, J. Pennington, J. Hron, K. Kenealy, K. Swersky, K. Mahajan, L. A Culp, L. Xiao, M. Bileschi, N. Constant, R. Novak, R. Liu, T. Warkentin, Y. Bansal, E. Dyer, B. Neyshabur, J. Sohl-Dickstein, and N. Fiedel · 2024
Closest in time.
V-dixmatch: A semi-supervised learning method for human action recognition in night video sensing
Chenxi Wang, Jingzhou Luo, Xing Luo, Haoran Qi, and Zhi Jin · 2024
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…