2020

LEEP: A New Measure to Evaluate Transferability of Learned Representations

Nguyen, Cuong V., Hassner, Tal, Seeger, Matthias et al.

Understand

We introduce a new measure to evaluate the transferability of representations learned by classifiers.

Our measure, the Log Expected Empirical Prediction (LEEP), is simple and easy to compute: when given a classifier trained on a source data set, it only requires running the target data set through this classifier once.
We analyze the properties of LEEP theoretically and demonstrate its effectiveness empirically.
Our analysis shows that LEEP can predict the performance and convergence speed of both transfer and meta-transfer learning methods, even for small or imbalanced data.

Built on

A connection between correlation and contingency
Hirschfeld, H. O · 1935
Earlier work this paper cites.
Das statistische problem der korrelation als variations-und eigenwertproblem und sein zusammenhang mit der ausgleichsrechnung
Gebelein, H · 1941
Earlier work this paper cites.
A stochastic approximation method
Robbins, H. and Monro, S · 1951
Earlier work this paper cites.
On measures of dependence
Rényi, A · 1959
Earlier work this paper cites.
Stochastic gradient learning in neural networks
Bottou, L · 1991
Earlier work this paper cites.
Exploiting task relatedness for multiple task learning
Ben-David, S. and Schuller, R · 2003
Earlier work this paper cites.
Detecting change in data streams
Kifer, D., Ben-David, S., and Gehrke, J · 2004
Earlier work this paper cites.
Analysis of representations for domain adaptation
Ben-David, S., Blitzer, J., Crammer, K., and Pereira, F · 2007
Earlier work this paper cites.
Modeling transfer relationships between learning tasks for improved inductive transfer
Eaton, E., Lane, T., et al · 2008
Earlier work this paper cites.
Learning Multiple Layers of Features from Tiny Images
Krizhevsky, A · 2009
Earlier work this paper cites.
Domain adaptation: Learning bounds and algorithms
Mansour, Y., Mohri, M., and Rostamizadeh, A · 2009
Earlier work this paper cites.
Adaptive transfer learning
Cao, B., Pan, S. J., Zhang, Y., Yeung, D.-Y., and Yang, Q · 2010
Earlier work this paper cites.
Analyzing the performance of multilayer neural networks for object recognition
Agrawal, P., Girshick, R., and Malik, J · 2014
Earlier work this paper cites.
An automated measure of MDP similarity for transfer in reinforcement learning
Ammar, H. B., Eaton, E., Taylor, M. E., Mocanu, D. C., Driessens, K., Weiss, G., and Tuyls, K · 2014
Earlier work this paper cites.
Return of the devil in the details: Delving deep into convolutional nets
Chatfield, K., Simonyan, K., Vedaldi, A., and Zisserman, A · 2014
Earlier work this paper cites.
DeCAF: A deep convolutional activation feature for generic visual recognition
Donahue, J., Jia, Y., Vinyals, O., Hoffman, J., Zhang, N., Tzeng, E., and Darrell, T · 2014
Earlier work this paper cites.
Rich feature hierarchies for accurate object detection and semantic segmentation
Girshick, R., Donahue, J., Darrell, T., and Malik, J · 2014
Earlier work this paper cites.
Stochastic gradient vb and the variational auto-encoder
Kingma, D. P. and Welling, M · 2014
Earlier work this paper cites.
Learning and transferring mid-level image representations using convolutional neural networks
Oquab, M., Bottou, L., Laptev, I., and Sivic, J · 2014
Earlier work this paper cites.

Similar

CNN features off-the-shelf: an astounding baseline for recognition
Razavian, A. S., Azizpour, H., Sullivan, J., and Carlsson, S · 2014
Cited alongside, same era.
How transferable are features in deep neural networks?
Yosinski, J., Clune, J., Bengio, Y., and Lipson, H · 2014
Cited alongside, same era.
Visualizing and understanding convolutional networks
Zeiler, M. D. and Fergus, R · 2014
Cited alongside, same era.
MXNet: A flexible and efficient machine learning library for heterogeneous distributed systems
Original
Chen, T., Li, M., Li, Y., Lin, M., Wang, N., Wang, M., Xiao, T., Xu, B., Zhang, C., and Zhang, Z · 2015
Cited alongside, same era.
Fully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., and Darrell, T · 2015
Cited alongside, same era.
Imagenet large scale visual recognition challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A. C., and Li, F.-F · 2015
Cited alongside, same era.
Learning inter-task transferability in the absence of target task samples
Sinapov, J., Narvekar, S., Leonetti, M., and Stone, P · 2015
Cited alongside, same era.
Transfer learning for class imbalance problems with inadequate data
Al-Stouhi, S. and Reddy, C. K · 2016
Cited alongside, same era.
Deep residual learning for image recognition
He, K., Zhang, X., Ren, S., and Sun, J · 2016
Cited alongside, same era.
Cross-stitch networks for multi-task learning
Misra, I., Shrivastava, A., Gupta, A., and Hebert, M · 2016
Cited alongside, same era.
Return of frustratingly easy domain adaptation
Sun, B., Feng, J., and Saenko, K · 2016
Cited alongside, same era.
A survey of transfer learning
Weiss, K., Khoshgoftaar, T. M., and Wang, D · 2016
Cited alongside, same era.

Then

Yolov3: An incremental improvement
Original
Redmon, J. and Farhadi, A · 2018
Later among the works it cites.
Taskonomy: Disentangling task transfer learning
Zamir, A. R., Sax, A., Shen, W., Guibas, L. J., Malik, J., and Savarese, S · 2018
Later among the works it cites.
Task2vec: Task embedding for meta-learning
Achille, A., Lam, M., Tewari, R., Ravichandran, A., Maji, S., Fowlkes, C. C., Soatto, S., and Perona, P · 2019
Later among the works it cites.
Regularized learning for domain adaptation under label shifts
Azizzadenesheli, K., Liu, A., Yang, F., and Anandkumar, A · 2019
Later among the works it cites.
An information-theoretic approach to transferability in task transfer learning
Bao, Y., Li, Y., Huang, S.-L., Zhang, L., Zheng, L., Zamir, A., and Guibas, L · 2019
Later among the works it cites.
P2L: Predicting transfer learning for images and semantic relations
Original
Bhattacharjee, B., Codella, N., Kender, J. R., Huo, S., Watson, P., Glass, M. R., Dube, P., Hill, M., and Belgodere, B · 2019
Later among the works it cites.
GluonCV and GluonNLP: Deep learning in computer vision and natural language processing
Original
Guo, J., He, H., He, T., Lausen, L., Li, M., Lin, H., Shi, X., Wang, C., Xie, J., Zha, S., et al · 2019
Later among the works it cites.
Dataset2vec: Learning dataset meta-features
Original
Jomaa, H. S., Grabocka, J., and Schmidt-Thieme, L · 2019
Later among the works it cites.
Do better imagenet models transfer better?
Kornblith, S., Shlens, J., and Le, Q. V · 2019
Later among the works it cites.
Toward understanding catastrophic forgetting in continual learning
Original
Nguyen, C. V., Achille, A., Lam, M., Hassner, T., Mahadevan, V., and Soatto, S · 2019
Later among the works it cites.
Fast and flexible multi-task classification using conditional neural adaptive processes
Requeima, J., Gordon, J., Bronskill, J., Nowozin, S., and Turner, R. E · 2019
Later among the works it cites.
Which tasks should be learned together in multi-task learning?
Original
Standley, T., Zamir, A. R., Chen, D., Guibas, L., Malik, J., and Savarese, S · 2019
Later among the works it cites.
Meta-transfer learning for few-shot learning
Sun, Q., Liu, Y., Chua, T.-S., and Schiele, B · 2019
Later among the works it cites.
Improving and understanding variational continual learning
Original
Swaroop, S., Nguyen, C. V., Bui, T. D., and Turner, R. E · 2019
Later among the works it cites.
Transferability and hardness of supervised classification tasks
Tran, A. T., Nguyen, C. V., and Hassner, T · 2019
Later among the works it cites.
FixyNN: Efficient hardware for mobile computer vision via transfer learning
Whatmough, P. N., Zhou, C., Hansen, P., Venkataramanaiah, S. K., Seo, J.-s., and Mattina, M · 2019
Later among the works it cites.
A baseline for few-shot image classification
Dhillon, G. S., Chaudhari, P., Ravichandran, A., and Soatto, S · 2020
Closest in time.
Meta-dataset: A dataset of datasets for learning to learn from few examples
Triantafillou, E., Zhu, T., Dumoulin, V., Lamblin, P., Evci, U., Xu, K., Goroshin, R., Gelada, C., Swersky, K., Manzagol, P.-A., and Larochelle, H · 2020
Closest in time.
Transfer learning
Yang, Q., Zhang, Y., Dai, W., and Pan, S. J · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…