2015

Knowledge Transfer Pre-training

Tang, Zhiyuan, Wang, Dong, Pan, Yiqiao et al.

Understand

Pre-training is crucial for learning deep neural networks.

Most of existing pre-training methods train simple models (e.g., restricted Boltzmann machines) and then stack them layer by layer to form the deep structure.
This layer-wise pre-training has found strong theoretical foundation and broad empirical support.
However, it is not easy to employ such method to pre-train models without a clear multi-layer structure,e.g., recurrent neural networks (RNNs).

Y. Bengio, P. Simard, and P. Frasconi, “Learning long-term dependencies with gradient descent is difficult,”
1994
Earlier work this paper cites.
G. E. Hinton and R. R. Salakhutdinov, “Reducing the dimensionality of data with neural networks,”
2006
Earlier work this paper cites.
Y. Bengio, P. Lamblin, D. Popovici, H. Larochelle
2007
Earlier work this paper cites.
D. Erhan, Y. Bengio, A. Courville, P.-A. Manzagol, P. Vincent, and S. Bengio, “Why does unsupervised pre-training help deep learning?”
2010
Earlier work this paper cites.
D. Yu, L. Deng, and G. Dahl, “Roles of pre-training and fine-tuning in context-dependent dbn-hmms for real-world speech recognition,” in
2010
Earlier work this paper cites.
G. E. Dahl, D. Yu, L. Deng, and A. Acero, “Large vocabulary continuous speech recognition with context-dependent DBN-HMMs,” in
2011
Earlier work this paper cites.
F. Seide, G. Li, X. Chen, and D. Yu, “Feature engineering in context-dependent deep neural networks for conversational speech transcription,” in
2011
Earlier work this paper cites.
G. Hinton, L. Deng, D. Yu, G. E. Dahl, A.-r. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T. N. Sainath
2012
Earlier work this paper cites.

G. E. Dahl, D. Yu, L. Deng, and A. Acero, “Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition,”
2012
Cited alongside, same era.
O. Vinyals, S. V. Ravuri, and D. Povey, “Revisiting recurrent neural networks for robust asr,” in
2012
Cited alongside, same era.
N. Boulanger-Lewandowski, Y. Bengio, and P. Vincent, “Modeling temporal dependencies in high-dimensional sequences: Application to polyphonic music generation and transcription,”
Original
2012
Cited alongside, same era.
L. Pasa, A. Testolin, and A. Sperduti, “A hmm-based pre-training approach for sequential data,” 2012
2012
Cited alongside, same era.
L. Deng and D. Yu, “Deep learning: Methods and applications,”
2013
Cited alongside, same era.
I. Sutskever, J. Martens, G. Dahl, and G. Hinton, “On the importance of initialization and momentum in deep learning,” in
2013
Cited alongside, same era.
A. Graves and N. Jaitly, “Towards end-to-end speech recognition with recurrent neural networks,” in
2014
Cited alongside, same era.
H. Sak, A. Senior, and F. Beaufays, “Long short-term memory recurrent neural network architectures for large scale acoustic modeling,” in
2014
Cited alongside, same era.
A. Paul and S. Venkatasubramanian, “Why does unsupervised deep learning work? - a perspective from group theory.”
Cited in the paper.

C. Weng, D. Yu, S. Watanabe, and B.-H. F. Juang, “Recurrent deep neural networks for robust speech recognition,” in
2014
Later among the works it cites.
L. Pasa and A. Sperduti, “Pre-training of recurrent neural networks via linear autoencoders,” pp. 3572–3580, 2014
2014
Later among the works it cites.
J. Ba and R. Caruana, “Do deep nets really need to be deep?” pp. 2654–2662, 2014
2014
Later among the works it cites.
G. E. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” in
2014
Later among the works it cites.
J. Li, R. Zhao, J.-T. Huang, and Y. Gong, “Learning small-size dnn with output-distribution-based criteria,” in
2014
Later among the works it cites.
A. Romero, N. Ballas, S. E. Kahou, A. Chassang, C. Gatta, and Y. Bengio, “Fitnets: Hints for thin deep nets,”
Original
2014
Later among the works it cites.
D. Yu and L. Deng, “Recurrent neural networks and related models,” in
2015
Closest in time.
W. Chan, N. R. Ke, and I. Lane, “Transferring knowledge from a rnn to a dnn,”
Original
2015
Closest in time.

alphaXiv searches the wider corpus for related work and actual follow-ups.

alphaXiv is searching for related work…

Reading the bibliography…