Fetching the paper…
Reading the bibliography…
Self-supervised representation learning methods aim to provide powerful deep feature learning without the requirement of large annotated datasets, thus alleviating the annotation bottleneck that is one of the main barriers to practical deployment of deep learning today.
K. M. Borgwardt, C. S. Ong, S. Schonauer, S. V. N. Vishwanathan, A. J. Smola, and H.-P. Kriegel, “Protein function prediction via graph kernels,” Bioinformatics , 2005
2005
Earlier work this paper cites.
R. Hadsell, S. Chopra, and Y. LeCun, “Dimensionality reduction by learning an invariant mapping,” in CVPR , 2006
2006
Earlier work this paper cites.
T. Hastie, R. Tibshirani, and J. Friedman, The elements of statistical learning: data mining, inference, and prediction , 2009
2009
Earlier work this paper cites.
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “ImageNet: A Large-Scale Hierarchical Image Database,” in CVPR , 2009
2009
Earlier work this paper cites.
M. Gutmann and A. Hyvärinen, “Noise-contrastive estimation: A new estimation principle for unnormalized statistical models,” Journal of Machine Learning Research , 2010
2010
Earlier work this paper cites.
U. von Luxburg, “Clustering stability: An overview,” Foundations and Trends in Machine Learning , vol. 2, no. 3, pp. 235–274, 2010
2010
Earlier work this paper cites.
T. Mikolov, I. Sutskever, K. Chen, G. Corrado, and J. Dean, “Distributed Representations of Words and Phrases and their Compositionality,” in NeurIPS , 2013
2013
Earlier work this paper cites.
A. Dosovitskiy, P. Fischer, J. T. Springenberg, M. Riedmiller, and T. Brox, “Discriminative Unsupervised Feature Learning with Exemplar Convolutional Neural Networks,” in NeurIPS , 2014
2014
Earlier work this paper cites.
C. Buck, K. Heafield, and B. v. Ooyen, “N-gram Counts and Language Models from the Common Crawl,” in LREC , 2014
2014
Earlier work this paper cites.
B. Thomee, D. A. Shamma, G. Friedland, B. Elizalde, K. Ni, D. Poland, D. Borth, and L.-J. Li, “YFCC100M: The New Data in Multimedia Research,” Communications of the ACM , 2015
2015
Earlier work this paper cites.
V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech: An ASR corpus based on public domain audio books,” in ICASSP , 2015
2015
Earlier work this paper cites.
K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in CVPR , 2016
2016
Earlier work this paper cites.
A. Grover and J. Leskovec, “node2vec: Scalable Feature Learning for Networks,” in KDD , 2016
2016
Earlier work this paper cites.
I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning , 2016
2016
Earlier work this paper cites.
D. Pathak, P. Krahenbuhl, J. Donahue, T. Darrell, and A. A. Efros, “Context Encoders: Feature Learning by Inpainting,” in CVPR , 2016
2016
Earlier work this paper cites.
G. Larsson, M. Maire, and G. Shakhnarovich, “Learning representations for automatic colorization,” in ECCV , 2016
2016
Earlier work this paper cites.
D. Amodei, S. Ananthanarayanan, R. Anubhai, J. Bai, E. Battenberg, C. Case, J. Casper, B. Catanzaro, Q. Cheng, G. Chen et al. , “Deep speech 2: End-to-end speech recognition in english and mandarin,” in ICML , 2016
2016
Earlier work this paper cites.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in NeurIPS , 2017
2017
Earlier work this paper cites.
C. Sun, A. Shrivastava, S. Singh, and A. Gupta, “Revisiting unreasonable effectiveness of data in deep learning era,” in ICCV , 2017
2017
Earlier work this paper cites.
W. L. Hamilton, R. Ying, and J. Leskovec, “Inductive Representation Learning on Large Graphs,” in NeurIPS , 2017
2017
Earlier work this paper cites.
W. Kay, J. Carreira, K. Simonyan, B. Zhang, C. Hillier, S. Vijayanarasimhan, F. Viola, T. Green, T. Back, P. Natsev, M. Suleyman, and A. Zisserman, “The Kinetics Human Action Video Dataset,” arXiv , 2017
2017
Earlier work this paper cites.
S. Merity, C. Xiong, J. Bradbury, and R. Socher, “Pointer Sentinel Mixture Models,” in ICLR , 2017
2017
Earlier work this paper cites.
J. F. Gemmeke, D. P. W. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter, “Audio Set: An ontology and human-labeled dataset for audio events,” in ICASSP , 2017
2017
Earlier work this paper cites.
T. N. Kipf and M. Welling, “Semi-Supervised Classification with Graph Convolutional Networks,” in ICLR , 2017
2017
Earlier work this paper cites.
M. Zitnik and J. Leskovec, “Predicting multicellular function through multi-layer tissue networks,” Bioinformatics , 2017
2017
Earlier work this paper cites.
A. v. d. Oord, Y. Li, and O. Vinyals, “Representation Learning with Contrastive Predictive Coding,” arXiv , 2018
2018
Earlier work this paper cites.
S. Gidaris, P. Singh, and N. Komodakis, “Unsupervised Representation Learning by Predicting Image Rotations,” ICLR , 2018
2018
Earlier work this paper cites.
D. Mahajan, R. Girshick, V. Ramanathan, K. He, M. Paluri, Y. Li, A. Bharambe, and L. van der Maaten, “Exploring the Limits of Weakly Supervised Pretraining,” in ECCV , 2018
2018
Earlier work this paper cites.
A. Owens and A. A. Efros, “Audio-Visual Scene Analysis with Self-Supervised Multisensory Features,” in ECCV , 2018
2018
Earlier work this paper cites.
M. Caron, P. Bojanowski, A. Joulin, and M. Douze, “Deep Clustering for Unsupervised Learning of Visual Features,” in ECCV , 2018
2018
Earlier work this paper cites.
M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, and L. Zettlemoyer, “Deep contextualized word representations,” in NAACL , 2018
2018
Earlier work this paper cites.
G. Van Horn, O. Mac Aodha, Y. Song, Y. Cui, C. Sun, A. Shepard, H. Adam, P. Perona, and S. Belongie, “The iNaturalist Species Classification and Detection Dataset,” in CVPR , 2018
2018
Earlier work this paper cites.
I. Golan and R. El-Yaniv, “Deep anomaly detection using geometric transformations,” in NeurIPS , 2018
2018
Earlier work this paper cites.
Z. Wu, B. Ramsundar, E. N. Feinberg, J. Gomes, C. Geniesse, A. S. Pappu, K. Leswing, and V. Pande, “MoleculeNet: A benchmark for molecular machine learning,” Chemical Science , 2018
2018
Cited alongside, same era.
D. Xu, J. Xiao, Z. Zhao, J. Shao, D. Xie, and Y. Zhuang, “Self-supervised Spatiotemporal Learning via Video Clip Order Prediction,” in CVPR , 2019
2019
Cited alongside, same era.
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” in NAACL , 2019
2019
Cited alongside, same era.
A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, “Language Models are Unsupervised Multitask Learners,” Tech. Rep., 2019
2019
Cited alongside, same era.
P. Veličković, W. Fedus, W. L. Hamilton, P. Liò, Y. Bengioy, and R. D. Hjelm, “Deep Graph Infomax,” in ICLR , 2019
2019
X. Jiao, Y. Yin, L. Shang, X. Jiang, X. Chen, L. Li, F. Wang, and Q. Liu, “TinyBERT: Distilling BERT for Natural Language Understanding,” in EMNLP , 2020
2020
Later among the works it cites.
T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei, “Language Models are Few-Shot Learners,” in NeurIPS , 2020
2020
Later among the works it cites.
P. Sarkar and A. Etemad, “Self-supervised Learning for ECG-based Emotion Recognition,” in ICASSP , 2020
2020
Later among the works it cites.
Y. Tian, D. Krishnan, and P. Isola, “Contrastive Multiview Coding,” in ECCV , 2020
2020
Later among the works it cites.
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Cited alongside, same era.
P. Goyal, D. Mahajan, A. Gupta, and I. Misra, “Scaling and Benchmarking Self-Supervised Visual Representation Learning,” in ICCV , 2019
2019
Cited alongside, same era.
A. Conneau and G. Lample, “Cross-lingual language model pretraining,” in NeurIPS , 2019
2019
Cited alongside, same era.
D. Hendrycks, M. Mazeika, S. Kadavath, and D. Song, “Using self-supervised learning can improve model robustness and uncertainty,” in NeurIPS , 2019
2019
Cited alongside, same era.
A. Kolesnikov, X. Zhai, and L. Beyer, “Revisiting Self-Supervised Visual Representation Learning,” in CVPR , 2019
2019
Cited alongside, same era.
K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick, “Momentum Contrast for Unsupervised Visual Representation Learning,” in CVPR , 2019
2019
Cited alongside, same era.
N. Saunshi, O. Plevrakis, S. Arora, M. Khodak, and H. Khandeparkar, “A theoretical analysis of contrastive unsupervised representation learning,” in ICML , 2019
2019
Cited alongside, same era.
J. Lu, D. Batra, D. Parikh, and S. Lee, “Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks,” in NeurIPS , 2019
2019
Cited alongside, same era.
Y. Tian, C. Sun, B. Poole, D. Krishnan, C. Schmid, and P. Isola, “What makes for good views for contrastive learning,” in NeurIPS , 2020
2020
Later among the works it cites.
C.-Y. Chuang, J. Robinson, Y.-C. Lin, A. Torralba, and S. Jegelka, “Debiased contrastive learning,” in NeurIPS , 2020
2020
Later among the works it cites.
X. Zhan, J. Xie, Z. Liu, Y. S. Ong, and C. C. Loy, “Online Deep Clustering for Unsupervised Representation Learning,” in CVPR , 2020
2020
Later among the works it cites.
2020
Later among the works it cites.
M. Chen, A. Radford, J. Wu, H. Jun, P. Dhariwal, D. Luan, and I. Sutskever, “Generative Pretraining From Pixels,” in ICML , 2020
2020
Later among the works it cites.
D. Luo, C. Liu, Y. Zhou, D. Yang, C. Ma, Q. Ye, and W. Wang, “Video Cloze Procedure for Self-Supervised Spatio-Temporal Learning,” in AAAI , 2020
2020
Later among the works it cites.
C. Tao, J. Qi, W. Lu, H. Wang, and H. Li, “Remote sensing image scene classification with self-supervised paradi under limited labeled samples,” 2020
2020
Later among the works it cites.
A. Taleb, W. Loetzsch, N. Danz, J. Severin, T. Gaertner, B. Bergner, and C. Lippert, “3d self-supervised methods for medical imaging,” NeurIPS , 2020
2020
Later among the works it cites.
T. Han, W. Xie, and A. Zisserman, “Self-supervised Co-Training for Video Representation Learning,” in NeurIPS , 2020
2020
Later among the works it cites.
J.-B. Alayrac, A. Recasens, R. Schneider, R. Arandjelovi, J. Ramapuram, J. De Fauw, L. Smaira, S. Dieleman, and A. Zisserman, “Self-Supervised MultiModal Versatile Networks,” in NeurIPS , 2020
2020
Later among the works it cites.
M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer, “BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension,” in ACL , 2020
2020
Later among the works it cites.
J. Y. Cheng, H. Goh, K. Dogrusoz, O. Tuzel, and E. A. Apple, “Subject-Aware Contrastive Learning for Biosignals,” arXiv , 2020
2020
Later among the works it cites.
N. Wu, B. Green, X. Ben, and S. O’Banion, “Deep transformer models for time series forecasting: The influenza prevalence case,” in ICML , 2020
2020
Later among the works it cites.
F.-Y. Sun, J. Hoffmann, V. Verma, and J. Tang, “InfoGraph: Unsupervised and Semi-supervised Graph-Level Representation Learning via Mutual Information Maximization,” in ICLR , 2020
2020
Later among the works it cites.
W. Hu, B. Liu, J. Gomes, M. Zitnik, P. Liang, V. Pande, and J. Leskovec, “Strategies for Pre-training Graph Neural Networks,” in ICLR , 2020
2020
Later among the works it cites.
R. Schwartz, J. Dodge, N. A. Smith, and O. Etzioni, “Green AI,” Communications of the ACM , 2020
2020
Later among the works it cites.
S. Purushwalkam and A. Gupta, “Demystifying Contrastive Self-Supervised Learning: Invariances, Augmentations and Dataset Biases,” in NeurIPS , 2020
2020
Later among the works it cites.
M. Rivière, A. Joulin, P.-E. Mazaré, and E. Dupoux, “Unsupervised pretraining transfers well across languages,” in ICASSP , 2020
2020
Later among the works it cites.
B. Zoph, G. Ghiasi, T.-Y. Lin, Y. Cui, H. Liu, E. D. Cubuk, and Q. Le, “Rethinking pre-training and self-training,” in NeurIPS , 2020
2020
Later among the works it cites.
M.-I. Georgescu, A. Barbalau, R. T. Ionescu, F. S. Khan, M. Popescu, and M. Shah, “Anomaly detection in video via self-supervised and multi-task learning,” CVPR , 2021
2021
Closest in time.
H. Gouk, T. M. Hospedales, and M. Pontil, “Distance-based regularisation of deep networks for fine-tuning,” in ICLR , 2021
2021
Closest in time.
L. Ericsson, H. Gouk, and T. M. Hospedales, “How Well Do Self-Supervised Models Transfer?” in CVPR , 2021
2021
Closest in time.
J. Zbontar, L. Jing, I. Misra, Y. LeCun, and S. Deny, “Barlow Twins: Self-Supervised Learning via Redundancy Reduction,” in ICML , 2021
2021
Closest in time.
A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, “Learning transferable visual models from natural language supervision,” arXiv , 2021
2021
Closest in time.
N. Saunshi, S. Malladi, and S. Arora, “A mathematical exploration of why language models help solve downstream tasks,” in ICLR , 2021
2021
Closest in time.
A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale,” in ICLR , 2021
2021
Closest in time.
O. Mac Aodha, “Benchmarking Representation Learning on Natural World Image Collections,” in CVPR , 2021
2021
Closest in time.
J. Du, E. Grave, B. Gunel, V. Chaudhary, O. Celebi, M. Auli, V. Stoyanov, and A. Conneau, “Self-training improves pre-training for natural language understanding,” in NACL , 2021
2021
Closest in time.