Fetching the paper…
Reading the bibliography…
Can we complete pre-training of Vision Transformers (ViT) without natural images and human-annotated labels? Although a pre-trained ViT seems to heavily rely on a large-scale dataset and human-annotated labels, recent large-scale datasets contain several problems in terms of privacy violations, inadequate fairness protection, and labor-intensive annotation.
The fractal geometry of nature
B. Mandelbrot · 1983
Earlier work this paper cites.
Fractals Everywhere
M. F. Barnsley · 1988
Earlier work this paper cites.
80 million tiny images: A large data set for nonparametric object and scene recognition
A. Torralba, R. Fergus, and W. T. Freeman · 2008
Earlier work this paper cites.
ImageNet: A Large-Scale Hierarchical Image Database
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei · 2009
Earlier work this paper cites.
ImageNet Classification with Deep Convolutional Neural Networks
A. Krizhevsky, Ilya Sutskever, and G E Hinton · 2012
Earlier work this paper cites.
Visualizing and understanding convolutional networks
M. D. Zeiler and R. Fergus · 2014
Earlier work this paper cites.
Unsupervised Visual Representation Learning by Context Prediction
C. Doersch, A. Gupta, and A. Efros · 2015
Earlier work this paper cites.
Imagenet large scale visual recognition challenge
O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei · 2015
Earlier work this paper cites.
Very Deep Convolutional Networks for Large-Scale Image Recognition
K. Simonyan and A. Zisserman · 2015
Earlier work this paper cites.
Going Deeper with Convolutions
C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich · 2015
Earlier work this paper cites.
Deep Residual Learning for Image Recognition
K. He, X. Zhang, S. Ren, and J. Sun · 2016
Earlier work this paper cites.
Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles
M. Noroozi and P. Favaro · 2016
Earlier work this paper cites.
Colorful Image Colorization
R. Zhang, P. Isola, and A. A. Efros · 2016
Earlier work this paper cites.
Densely Connected Convolutional Networks
G. Huang, Z. Liu, L. van der Maaten, and K. Q. Weinberger · 2017
Earlier work this paper cites.
Representation Learning by Learning to Count
M. Noroozi, H. Pirsiavash, and P. Favaro · 2017
Cited alongside, same era.
Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin · 2017
Cited alongside, same era.
Aggregated Residual Transformations for Deep Neural Networks
S. Xie, R. Girshick, P. Dollár, Z. Tu, and K. He · 2017
Cited alongside, same era.
Places: A 10 million Image Database for Scene Recognition
B. Zhou, A. Lapedriza, A. Khosla, A. Oliva, and A. Torralba · 2017
Cited alongside, same era.
Unsupervised Representation Learning by Predicting Image Rotations
S. Gidaris, P. Singh, and N. Komodakis · 2018
Cited alongside, same era.
Exploring the Limits of Weakly Supervised Pretraining
D. Mahajan, R. Girshick, V. Ranathan, K. He, M. Paluri, Y. Li, A. Bharambe, and L. van der Maaten · 2018
Cited alongside, same era.
Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
M. Caron, I. Misra, J. Mairal, P. Goyal, P. Bojanowski, and A. Joulin · 2020
Later among the works it cites.
A Simple Framework for Contrastive Learning of Visual Representations
T. Chen, S. Kornblith, M. Norouzi, and G. Hinton · 2020
Later among the works it cites.
Big self-supervised models are strong semi-supervised learners
T. Chen, S. Kornblith, K. Swersky, M. Norouzi, and G. Hinton · 2020
Later among the works it cites.
Improved baselines with momentum contrastive learning
X. Chen, H. Fan, R. Girshick, and K. He · 2020
Later among the works it cites.
Momentum Contrast for Unsupervised Visual Representation Learning
K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick · 2020
Later among the works it cites.
Pre-training without Natural Images
H. Kataoka, K. Okayasu, A. Matsumoto, E. Yamagata, R. Yamada, N. Inoue, A. Nakamura, and Y. Satoh · 2020
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Boosting Self-Supervised Learning via Knowledge Transfer
M. Noroozi, A. Vinjimoor, P. Favaro, and H. Pirsiavash · 2018
Cited alongside, same era.
Improving language understanding by generative pre-training
A. Radford, K. Narasimhan, T Salimans, and I. Sutskever · 2018
Cited alongside, same era.
Language Models are Unsupervised Multitask Learners
A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever · 2018
Cited alongside, same era.
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova · 2019
Cited alongside, same era.
Rethinking ImageNet Pre-training
K. He, R. Girshick, and P. Dollár · 2019
Cited alongside, same era.
EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
M. Tan and Q. V. Le · 2019
Cited alongside, same era.
Later among the works it cites.
Big Transfer (BiT): General Visual Representation Learning
A. Kolesnikov, L. Beyer, X. Zhai, J. Puigcerver, J. Yung, S. Gelly, and N. Houlsby · 2020
Later among the works it cites.
Scaling Laws for Neural Language Models
J. Laplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei · 2020
Later among the works it cites.
Meta Pseudo Labels
H. Pham, Z. Dai, Q. Xie, M.-T. Luong, and Q. V. Le · 2020
Later among the works it cites.
Training data-efficient image transformers & distillation through attention
H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jégou · 2020
Later among the works it cites.
Towards Fairer Datasets: Filtering and Balancing the Distribution of the People Subtree in the ImageNet Hierarchy
K. Yang, K. Qinami, L. Fei-Fei, J. Deng, and O. Russakovsky · 2020
Later among the works it cites.
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby · 2021
Closest in time.
Sharpness-aware Minimization for Efficiently Improving Generalization
P. Foret, A. Kleiner, H. Mobahi, and B. Neyshabur · 2021
Closest in time.