Built on

X. Wang and J. Schneider, “Flexible transfer learning under support and model shift,” in Advances in Neural Information Processing Systems , 2014, pp. 1898–1906
1906
Earlier work this paper cites.
Y. LeCun, J. S. Denker, and S. A. Solla, “Optimal brain damage,” in Advances in Neural Information Processing Systems 2 , D. S. Touretzky, Ed. Morgan-Kaufmann, 1990, pp. 598–605. [Online]. Available: http://papers.nips.cc/paper/250-optimal-brain-damage.pdf
1990
Earlier work this paper cites.
B. Hassibi, D. G. Stork et al. , “Second order derivatives for network pruning: Optimal brain surgeon,” Advances in neural information processing systems , pp. 164–164, 1993
1993
Earlier work this paper cites.
Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” Proceedings of the IEEE , vol. 86, no. 11, pp. 2278–2324, 1998
1998
Earlier work this paper cites.
C. Buciluǎ, R. Caruana, and A. Niculescu-Mizil, “Model compression,” in Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining . ACM, 2006, pp. 535–541
2006
Earlier work this paper cites.
K. Kim, S. Lee, J.-Y. Kim, M. Kim, and H.-J. Yoo, “A configurable heterogeneous multicore architecture with cellular neural network for real-time object recognition,” IEEE Transactions on Circuits and Systems for Video Technology , vol. 19, no. 11, pp. 1612–1622, 2009
2009
Earlier work this paper cites.
D. Erhan, P.-A. Manzagol, Y. Bengio, S. Bengio, and P. Vincent, “The difficulty of training deep architectures and the effect of unsupervised pre-training.” in AISTATS , vol. 5, 2009, pp. 153–160
2009
Earlier work this paper cites.
A. Krizhevsky and G. Hinton, “Learning multiple layers of features from tiny images,” Master’s thesis, Department of Computer Science, University of Toronto , 2009
2009
Earlier work this paper cites.
S. J. Pan and Q. Yang, “A survey on transfer learning,” IEEE Transactions on knowledge and data engineering , vol. 22, no. 10, pp. 1345–1359, 2010
2010
Earlier work this paper cites.
M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman, “The pascal visual object classes (voc) challenge,” International journal of computer vision , vol. 88, no. 2, pp. 303–338, 2010
2010
Earlier work this paper cites.
S. J. Pan, I. W. Tsang, J. T. Kwok, and Q. Yang, “Domain adaptation via transfer component analysis,” IEEE Transactions on Neural Networks , vol. 22, no. 2, pp. 199–210, 2011
2011
Earlier work this paper cites.
N. Sudha, A. Mohan, and P. K. Meher, “A self-configurable systolic architecture for face recognition system based on principal component neural network,” IEEE transactions on circuits and systems for video technology , vol. 21, no. 8, pp. 1071–1084, 2011
2011
Earlier work this paper cites.
D. C. Cireşan, U. Meier, J. Masci, L. M. Gambardella, and J. Schmidhuber, “High-performance neural networks for visual object classification,” arXiv preprint arXiv:1102.0183 , 2011
Original
2011
Earlier work this paper cites.
G. E. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever, and R. R. Salakhutdinov, “Improving neural networks by preventing co-adaptation of feature detectors,” arXiv preprint arXiv:1207.0580 , 2012
Original
2012
Earlier work this paper cites.
M. Denil, B. Shakibi, L. Dinh, N. de Freitas et al. , “Predicting parameters in deep learning,” in Advances in Neural Information Processing Systems , 2013, pp. 2148–2156
2013
Earlier work this paper cites.
K. Zhang, B. Schölkopf, K. Muandet, and Z. Wang, “Domain adaptation under target and conditional shift.” in ICML (3) , 2013, pp. 819–827
2013
Earlier work this paper cites.
I. J. Goodfellow, D. Warde-Farley, M. Mirza, A. C. Courville, and Y. Bengio, “Maxout networks.” ICML (3) , vol. 28, pp. 1319–1327, 2013
2013
Earlier work this paper cites.
T. Lin, M. Maire, S. J. Belongie, L. D. Bourdev, R. B. Girshick, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft COCO: common objects in context,” CoRR , vol. abs/1405.0312, 2014. [Online]. Available: http://arxiv.org/abs/1405.0312
Original
2014
Earlier work this paper cites.
M. Jaderberg, A. Vedaldi, and A. Zisserman, “Speeding up convolutional neural networks with low rank expansions,” arXiv preprint arXiv:1405.3866 , 2014
Original
2014
Earlier work this paper cites.

Similar

A. Romero, N. Ballas, S. E. Kahou, A. Chassang, C. Gatta, and Y. Bengio, “Fitnets: Hints for thin deep nets,” arXiv preprint arXiv:1412.6550 , 2014
Original
2014
Cited alongside, same era.
Y. Ganin and V. Lempitsky, “Unsupervised domain adaptation by backpropagation,” arXiv preprint arXiv:1409.7495 , 2014
Original
2014
Cited alongside, same era.
M. D. Zeiler and R. Fergus, “Visualizing and understanding convolutional networks,” in European conference on computer vision . Springer, 2014, pp. 818–833
2014
Cited alongside, same era.
M. Oquab, L. Bottou, I. Laptev, and J. Sivic, “Learning and transferring mid-level image representations using convolutional neural networks,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2014, pp. 1717–1724
2014
Cited alongside, same era.

Then

C.-Y. Lee, S. Xie, P. W. Gallagher, Z. Zhang, and Z. Tu, “Deeply-supervised nets.” in AISTATS , vol. 2, no. 3, 2015, p. 5
2015
Later among the works it cites.
C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2015, pp. 1–9
2015
Later among the works it cites.
S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” arXiv preprint arXiv:1502.03167 , 2015
Original
2015
Later among the works it cites.
K. Lenc and A. Vedaldi, “R-cnn minus r,” arXiv preprint arXiv:1506.06981 , 2015
Original
2015
Later among the works it cites.
T. Chen, M. Li, Y. Li, M. Lin, N. Wang, M. Wang, T. Xiao, B. Xu, C. Zhang, and Z. Zhang, “Mxnet: A flexible and efficient machine learning library for heterogeneous distributed systems,” arXiv preprint arXiv:1512.01274 , 2015

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…

2017

Knowledge Projection for Deep Neural Networks

Zhang, Zhi, Ning, Guanghan, He, Zhihai

Understand

While deeper and wider neural networks are actively pushing the performance limits of various computer vision and machine learning tasks, they often require large sets of labeled data for effective training and suffer from extremely high computational complexity.

In this paper, we will develop a new framework for training deep neural networks on datasets with limited labeled samples using cross-network knowledge projection which is able to improve the network performance while reducing the overall computational complexity significantly.
Specifically, a large pre-trained teacher network is used to observe samples from the training data.
A projection matrix is learned to project this teacher-level knowledge and its visual representations from an intermediate layer of the teacher network to an intermediate layer of a thinner and faster student network to guide and regulate its training process.

Built on

X. Wang and J. Schneider, “Flexible transfer learning under support and model shift,” in Advances in Neural Information Processing Systems , 2014, pp. 1898–1906
1906
Earlier work this paper cites.
Y. LeCun, J. S. Denker, and S. A. Solla, “Optimal brain damage,” in Advances in Neural Information Processing Systems 2 , D. S. Touretzky, Ed. Morgan-Kaufmann, 1990, pp. 598–605. [Online]. Available: http://papers.nips.cc/paper/250-optimal-brain-damage.pdf
1990
Earlier work this paper cites.
B. Hassibi, D. G. Stork et al. , “Second order derivatives for network pruning: Optimal brain surgeon,” Advances in neural information processing systems , pp. 164–164, 1993
1993
Earlier work this paper cites.
Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” Proceedings of the IEEE , vol. 86, no. 11, pp. 2278–2324, 1998
1998
Earlier work this paper cites.
C. Buciluǎ, R. Caruana, and A. Niculescu-Mizil, “Model compression,” in Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining . ACM, 2006, pp. 535–541
2006
Earlier work this paper cites.
K. Kim, S. Lee, J.-Y. Kim, M. Kim, and H.-J. Yoo, “A configurable heterogeneous multicore architecture with cellular neural network for real-time object recognition,” IEEE Transactions on Circuits and Systems for Video Technology , vol. 19, no. 11, pp. 1612–1622, 2009
2009
Earlier work this paper cites.
D. Erhan, P.-A. Manzagol, Y. Bengio, S. Bengio, and P. Vincent, “The difficulty of training deep architectures and the effect of unsupervised pre-training.” in AISTATS , vol. 5, 2009, pp. 153–160
2009
Earlier work this paper cites.
A. Krizhevsky and G. Hinton, “Learning multiple layers of features from tiny images,” Master’s thesis, Department of Computer Science, University of Toronto , 2009
2009
Earlier work this paper cites.
S. J. Pan and Q. Yang, “A survey on transfer learning,” IEEE Transactions on knowledge and data engineering , vol. 22, no. 10, pp. 1345–1359, 2010
2010
Earlier work this paper cites.
M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman, “The pascal visual object classes (voc) challenge,” International journal of computer vision , vol. 88, no. 2, pp. 303–338, 2010
2010
Earlier work this paper cites.
S. J. Pan, I. W. Tsang, J. T. Kwok, and Q. Yang, “Domain adaptation via transfer component analysis,” IEEE Transactions on Neural Networks , vol. 22, no. 2, pp. 199–210, 2011
2011
Earlier work this paper cites.
N. Sudha, A. Mohan, and P. K. Meher, “A self-configurable systolic architecture for face recognition system based on principal component neural network,” IEEE transactions on circuits and systems for video technology , vol. 21, no. 8, pp. 1071–1084, 2011
2011
Earlier work this paper cites.
D. C. Cireşan, U. Meier, J. Masci, L. M. Gambardella, and J. Schmidhuber, “High-performance neural networks for visual object classification,” arXiv preprint arXiv:1102.0183 , 2011
Original
2011
Earlier work this paper cites.
G. E. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever, and R. R. Salakhutdinov, “Improving neural networks by preventing co-adaptation of feature detectors,” arXiv preprint arXiv:1207.0580 , 2012
Original
2012
Earlier work this paper cites.
M. Denil, B. Shakibi, L. Dinh, N. de Freitas et al. , “Predicting parameters in deep learning,” in Advances in Neural Information Processing Systems , 2013, pp. 2148–2156
2013
Earlier work this paper cites.
K. Zhang, B. Schölkopf, K. Muandet, and Z. Wang, “Domain adaptation under target and conditional shift.” in ICML (3) , 2013, pp. 819–827
2013
Earlier work this paper cites.
I. J. Goodfellow, D. Warde-Farley, M. Mirza, A. C. Courville, and Y. Bengio, “Maxout networks.” ICML (3) , vol. 28, pp. 1319–1327, 2013
2013
Earlier work this paper cites.
T. Lin, M. Maire, S. J. Belongie, L. D. Bourdev, R. B. Girshick, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft COCO: common objects in context,” CoRR , vol. abs/1405.0312, 2014. [Online]. Available: http://arxiv.org/abs/1405.0312
Original
2014
Earlier work this paper cites.
M. Jaderberg, A. Vedaldi, and A. Zisserman, “Speeding up convolutional neural networks with low rank expansions,” arXiv preprint arXiv:1405.3866 , 2014
Original
2014
Earlier work this paper cites.

Similar

A. Romero, N. Ballas, S. E. Kahou, A. Chassang, C. Gatta, and Y. Bengio, “Fitnets: Hints for thin deep nets,” arXiv preprint arXiv:1412.6550 , 2014
Original
2014
Cited alongside, same era.
Y. Ganin and V. Lempitsky, “Unsupervised domain adaptation by backpropagation,” arXiv preprint arXiv:1409.7495 , 2014
Original
2014
Cited alongside, same era.
M. D. Zeiler and R. Fergus, “Visualizing and understanding convolutional networks,” in European conference on computer vision . Springer, 2014, pp. 818–833
2014
Cited alongside, same era.
M. Oquab, L. Bottou, I. Laptev, and J. Sivic, “Learning and transferring mid-level image representations using convolutional neural networks,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2014, pp. 1717–1724
2014
Cited alongside, same era.

Then

C.-Y. Lee, S. Xie, P. W. Gallagher, Z. Zhang, and Z. Tu, “Deeply-supervised nets.” in AISTATS , vol. 2, no. 3, 2015, p. 5
2015
Later among the works it cites.
C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2015, pp. 1–9
2015
Later among the works it cites.
S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” arXiv preprint arXiv:1502.03167 , 2015
Original
2015
Later among the works it cites.
K. Lenc and A. Vedaldi, “R-cnn minus r,” arXiv preprint arXiv:1506.06981 , 2015
Original
2015
Later among the works it cites.
T. Chen, M. Li, Y. Li, M. Lin, N. Wang, M. Wang, T. Xiao, B. Xu, C. Zhang, and Z. Zhang, “Mxnet: A flexible and efficient machine learning library for heterogeneous distributed systems,” arXiv preprint arXiv:1512.01274 , 2015

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

J. Yosinski, J. Clune, Y. Bengio, and H. Lipson, “How transferable are features in deep neural networks?” in Advances in neural information processing systems , 2014, pp. 3320–3328

2014

Cited alongside, same era.

H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, and M. Marchand, “Domain-adversarial neural networks,” arXiv preprint arXiv:1412.4446 , 2014

Original

2014

Cited alongside, same era.

M. Ghifary, W. B. Kleijn, and M. Zhang, “Domain adaptive neural networks for object recognition,” in Pacific Rim International Conference on Artificial Intelligence . Springer, 2014, pp. 898–904

2014

Cited alongside, same era.

K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” arXiv preprint arXiv:1409.1556 , 2014

Original

2014

Cited alongside, same era.

E. L. Denton, W. Zaremba, J. Bruna, Y. LeCun, and R. Fergus, “Exploiting linear structure within convolutional networks for efficient evaluation,” in Advances in Neural Information Processing Systems , 2014, pp. 1269–1277

2014

Cited alongside, same era.

Y. Gong, L. Liu, M. Yang, and L. Bourdev, “Compressing deep convolutional networks using vector quantization,” arXiv preprint arXiv:1412.6115 , 2014

Original

2014

Cited alongside, same era.

J. Ba and R. Caruana, “Do deep nets really need to be deep?” in Advances in neural information processing systems , 2014, pp. 2654–2662

2014

Cited alongside, same era.

J. T. Springenberg, A. Dosovitskiy, T. Brox, and M. Riedmiller, “Striving for simplicity: The all convolutional net,” arXiv preprint arXiv:1412.6806 , 2014

Original

2014

Cited alongside, same era.

Original

2015