2017

ProjectionNet: Learning Efficient On-Device Deep Networks Using Neural Projections

Ravi, Sujith

Understand

Deep neural networks have become ubiquitous for applications related to visual recognition and language understanding tasks.

However, it is often prohibitive to use typical neural networks on devices like mobile phones or smart watches since the model sizes are huge and cannot fit in the limited memory available on such devices.
While these devices could make use of machine learning models running on high-performance data centers with CPUs or GPUs, this is not feasible for many applications because data can be privacy sensitive and inference needs to be performed directly "on" device.
We introduce a new architecture for training compact neural networks using a joint optimization framework.

Built on

A. Stolcke, “Entropy-based pruning of backoff language models,” CoRR
2000
Earlier work this paper cites.
M. S. Charikar, “Similarity estimation techniques from rounding algorithms,” in Proceedings of the Thiry-fourth Annual ACM Symposium on Theory of Computing
2002
Earlier work this paper cites.
K. Ganchev and M. Dredze, “Small Statistical Models by Random Feature Mixing,” in Proceedings of the ACL-08: HLT Workshop on Mobile Language Processing
2008
Earlier work this paper cites.
New York, NY, USA: Cambridge University Press, 2008
C. D. Manning, P. Raghavan, and H. Schütze, Introduction to Information Retrieval · 2008
Earlier work this paper cites.
B.-G. Chun and P. Maniatis, “Augmented smartphone applications through clone cloud execution,” in Proceedings of the 12th Conference on Hot Topics in Operating Systems
2009
Earlier work this paper cites.
K. Weinberger, A. Dasgupta, J. Langford, A. Smola, and J. Attenberg, “Feature hashing for large scale multitask learning,” in Proceedings of the 26th Annual International Conference on Machine Learning
2009
Earlier work this paper cites.
Q. Shi, J. Petterson, G. Dror, J. Langford, A. Smola, and S. Vishwanathan, “Hash kernels for structured data,” J. Mach. Learn. Res
2009
Earlier work this paper cites.
M. Schuster, “Speech recognition for mobile devices at google,” in Proceedings of the 11th Pacific Rim International Conference on Trends in Artificial Intelligence
2010
Earlier work this paper cites.
V. Nair and G. E. Hinton, “Rectified linear units improve restricted boltzmann machines,” in Proceedings of the 27th International Conference on Machine Learning (ICML-10)
2010
Earlier work this paper cites.
A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in Advances in Neural Information Processing Systems
2012
Earlier work this paper cites.

Similar

G. Hinton, L. Deng, D. Yu, G. E. Dahl, A.-r. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T. N. Sainath, et al
2012
Cited alongside, same era.
K. H. Lee and N. Verma, “A low-power processor with configurable embedded machine-learning accelerators for high-order and adaptive analysis of medical-sensor signals,” IEEE Journal of Solid-State Circuits
2013
Cited alongside, same era.
M. Denil, B. Shakibi, L. Dinh, M. Ranzato, and N. de Freitas, “Predicting parameters in deep learning,” in Proceedings of the 26th International Conference on Neural Information Processing Systems
2013
Cited alongside, same era.
T. Mikolov, K. Chen, G. Corrado, and J. Dean, “Efficient estimation of word representations in vector space,” CoRR
Original
2013
Cited alongside, same era.
I. Sutskever, O. Vinyals, and Q. V. Le, “Sequence to sequence learning with neural networks,” in Advances in Neural Information Processing Systems
2014
Cited alongside, same era.
D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation by jointly learning to align and translate,” CoRR
Original
2014
Cited alongside, same era.
M. Courbariaux, Y. Bengio, and J. David, “Low precision arithmetic for deep learning,” CoRR
Original
2014
Cited alongside, same era.
Y. Gong, L. Liu, M. Yang, and L. D. Bourdev, “Compressing deep convolutional networks using vector quantization,” CoRR
Original
2014
Cited alongside, same era.
N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Dropout: A simple way to prevent neural networks from overfitting,” Journal of Machine Learning Research
2014
Cited alongside, same era.
I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative Adversarial Networks,” ArXiv e-prints
2014
Cited alongside, same era.
“Android Wear 2.0: Make the most of every minute.” https://blog.google/products/android-wear/android-wear-20-make-most-every-minute/
Cited in the paper.
“The MNIST database of handwritten digits.” http://yann.lecun.com/exdb/mnist/
Cited in the paper.

Then

W. Chen, J. T. Wilson, S. Tyree, K. Q. Weinberger, and Y. Chen, “Compressing neural networks with the hashing trick,” in Proceedings of the 32nd International Conference on International Conference on Machine Learning - Volume 37
2015
Later among the works it cites.
G. Hinton, O. Vinyals, and J. Dean, “Distilling the Knowledge in a Neural Network,” ArXiv e-prints
2015
Later among the works it cites.
J. Wang, W. Liu, S. Kumar, and S. Chang, “Learning to hash for indexing big data - A survey,” CoRR
Original
2015
Later among the works it cites.
Software available from tensorflow.org
M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, S. Ghemawat, I. Goodfellow, A. Harp, G. Irving, M. Isard, Y. Jia, R. Jozefowicz, L. Kaiser, M. Kudlur, J. Levenberg, D. Mané, R. Monga, S. Moore, D. Murray, C. Olah, M. Schuster, J. Shlens, B. Steiner, I. Sutskever, K. Talwar, P. Tucker, V. Vanhoucke, V. Vasudevan, F. Viégas, O. Vinyals, P. Warden, M. Wattenberg, M. Wicke, Y. Yu, and X. Zheng, “TensorFlow: Large-scale machine learning on heterogeneous systems,” 2015 · 2015
Later among the works it cites.
A. Kannan, K. Kurach, S. Ravi, T. Kaufmann, A. Tomkins, B. Miklos, G. Corrado, L. Lukacs, M. Ganea, P. Young, and V. Ramavajjala, “Smart reply: Automated response suggestion for email,” in Proceedings of the ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD)
2016
Later among the works it cites.
M. Courbariaux and Y. Bengio, “Binarynet: Training deep neural networks with weights and activations constrained to +1 or -1,” CoRR
Original
2016
Later among the works it cites.
S. Ravi and Q. Diao, “Large scale distributed semi-supervised learning using streaming approximation,” in Proceedings of the 19th International Conference on Artificial Intelligence and Statistics
2016
Later among the works it cites.
Z. Yang, W. W. Cohen, and R. Salakhutdinov, “Revisiting semi-supervised learning with graph embeddings,” CoRR
Original
2016
Later among the works it cites.
J. Konecný, H. B. McMahan, D. Ramage, and P. Richtárik, “Federated optimization: Distributed machine learning for on-device intelligence,” CoRR
Original
2016
Later among the works it cites.
T. D. Bui, S. Ravi, and V. Ramavajjala, “Neural graph machines: Learning neural networks using graphs,” CoRR
Original
2017
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…