2019

A Baseline for Few-Shot Image Classification

Dhillon, Guneet S., Chaudhari, Pratik, Ravichandran, Avinash et al.

Understand

Fine-tuning a deep network trained with the standard cross-entropy loss is a strong baseline for few-shot learning.

When fine-tuned transductively, this outperforms the current state-of-the-art on standard datasets such as Mini-ImageNet, Tiered-ImageNet, CIFAR-FS and FC-100 with the same hyper-parameters.
The simplicity of this approach enables us to demonstrate the first few-shot learning results on the ImageNet-21k dataset.
We find that using a large number of meta-training classes results in high few-shot accuracies even for a large number of few-shot classes.

Built on

Shift of bias for inductive concept learning
Paul E Utgoff · 1986
Earlier work this paper cites.
Evolutionary principles in self-referential learning
Jurgen Schmidhuber · 1987
Earlier work this paper cites.
On the optimization of a synaptic learning rule
Samy Bengio, Yoshua Bengio, Jocelyn Cloutier, and Jan Gecsei · 1992
Earlier work this paper cites.
Signature verification using a” siamese” time delay neural network
Jane Bromley, Isabelle Guyon, Yann LeCun, Eduard Säckinger, and Roopak Shah · 1994
Earlier work this paper cites.
Learning internal representations
Jonathan Baxter · 1995
Earlier work this paper cites.
Lifelong learning algorithms
Sebastian Thrun · 1998
Earlier work this paper cites.
Transductive inference for text classification using support vector machines
Thorsten Joachims · 1999
Earlier work this paper cites.
Learning to learn using gradient descent
Sepp Hochreiter, A Steven Younger, and Peter R Conwell · 2001
Earlier work this paper cites.
Learning with local and global consistency
Dengyong Zhou, Olivier Bousquet, Thomas N Lal, Jason Weston, and Bernhard Schölkopf · 2004
Earlier work this paper cites.
Learning a similarity metric discriminatively, with application to face verification
Sumit Chopra, Raia Hadsell, Yann LeCun, et al · 2005
Earlier work this paper cites.
Semi-supervised learning by entropy minimization
Yves Grandvalet and Yoshua Bengio · 2005
Earlier work this paper cites.
Visualizing data using t-SNE
Laurens van der Maaten and Geoffrey Hinton · 2008
Earlier work this paper cites.
Imagenet: A large-scale hierarchical image database
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei · 2009
Earlier work this paper cites.
Learning multiple layers of features from tiny images
Alex Krizhevsky and Geoffrey Hinton · 2009
Earlier work this paper cites.
The nature of statistical learning theory
Vladimir Vapnik · 2013
Earlier work this paper cites.
Adam: A method for stochastic optimization
Original
Diederik P Kingma and Jimmy Ba · 2014
Earlier work this paper cites.
Deep transfer metric learning
Junlin Hu, Jiwen Lu, and Yap-Peng Tan · 2015
Earlier work this paper cites.

Similar

Batch normalization: Accelerating deep network training by reducing internal covariate shift
Original
Sergey Ioffe and Christian Szegedy · 2015
Cited alongside, same era.
Gradient-based hyperparameter optimization through reversible learning
Dougal Maclaurin, David Duvenaud, and Ryan Adams · 2015
Cited alongside, same era.
Distributional smoothing with virtual adversarial training
Original
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, Ken Nakae, and Shin Ishii · 2015
Cited alongside, same era.
Semi-supervised classification with graph convolutional networks
Original
Thomas N Kipf and Max Welling · 2016
Cited alongside, same era.
Sgdr: Stochastic gradient descent with warm restarts
Original
Ilya Loshchilov and Frank Hutter · 2016
Cited alongside, same era.
Optimization as a model for few-shot learning
Sachin Ravi and Hugo Larochelle · 2016
Cited alongside, same era.
Regularization with stochastic transformations and perturbations for deep semi-supervised learning
Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen · 2016
Cited alongside, same era.
Rethinking the inception architecture for computer vision
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna · 2016
Cited alongside, same era.
Matching networks for one shot learning
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al · 2016
Cited alongside, same era.
Wide residual networks
Original
Sergey Zagoruyko and Nikos Komodakis · 2016
Cited alongside, same era.
Good semi-supervised learning that requires a bad gan
Zihang Dai, Zhilin Yang, Fan Yang, William W Cohen, and Ruslan R Salakhutdinov · 2017
Cited alongside, same era.
Model-agnostic meta-learning for fast adaptation of deep networks
Chelsea Finn, Pieter Abbeel, and Sergey Levine · 2017
Cited alongside, same era.

Then

mixup: Beyond empirical risk minimization
Original
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz · 2017
Later among the works it cites.
Variadic learning by bayesian nonparametric deep embedding
Kelsey R Allen, Hanul Shin, Evan Shelhamer, and Josh B Tenenbaum · 2018
Later among the works it cites.
Meta-learning with differentiable closed-form solvers
Original
Luca Bertinetto, João F Henriques, Philip HS Torr, and Andrea Vedaldi · 2018
Later among the works it cites.
A closer look at few-shot classification
Wei-Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang · 2018
Later among the works it cites.
Dynamic few-shot visual learning without forgetting
Spyros Gidaris and Nikos Komodakis · 2018
Later among the works it cites.
Highly scalable deep learning training system with mixed-precision: Training imagenet in four minutes
Original
Xianyan Jia, Shutao Song, Wei He, Yangzihao Wang, Haidong Rong, Feihu Zhou, Liqiang Xie, Zhenyu Guo, Yuanzhou Yang, Liwei Yu, et al · 2018
Later among the works it cites.
On first-order meta-learning algorithms
Original
Alex Nichol, Joshua Achiam, and John Schulman · 2018
Later among the works it cites.
Tadam: Task dependent adaptive metric for improved few-shot learning
Boris Oreshkin, Pau Rodríguez López, and Alexandre Lacoste · 2018
Later among the works it cites.
Low-shot learning with imprinted weights
Hang Qi, Matthew Brown, and David G Lowe · 2018
Later among the works it cites.
Few-shot image recognition by predicting parameters from activations
Siyuan Qiao, Chenxi Liu, Wei Shen, and Alan L Yuille · 2018
Later among the works it cites.
Meta-learning for semi-supervised few-shot classification
Original
Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua B Tenenbaum, Hugo Larochelle, and Richard S Zemel · 2018
Later among the works it cites.
Meta-learning with latent embedding optimization
Original
Andrei A Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell · 2018
Later among the works it cites.
Bag of tricks for image classification with convolutional neural networks
Original
Junyuan Xie, Tong He, Zhi Zhang, Hang Zhang, Zhongyue Zhang, and Mu Li · 2018
Later among the works it cites.
Analyzing and improving representations with the soft nearest neighbor loss
Original
Nicholas Frosst, Nicolas Papernot, and Geoffrey Hinton · 2019
Closest in time.
Meta-learning with differentiable convex optimization
Original
Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto · 2019
Closest in time.
Few-shot learning with embedded class models and shot-free meta training, 2019
Avinash Ravichandran, Rahul Bhotika, and Stefano Soatto · 2019
Closest in time.
Meta-dataset: A dataset of datasets for learning to learn from few examples
Original
Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Kelvin Xu, Ross Goroshin, Carles Gelada, Kevin Swersky, Pierre-Antoine Manzagol, and Hugo Larochelle · 2019
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…