Fetching the paper…
Reading the bibliography…
Recent research has shown that training low-rank neural networks can effectively reduce the total number of trainable parameters without sacrificing predictive accuracy, resulting in end-to-end speedups.
Imagenet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L · 2009
Earlier work this paper cites.
Learning multiple layers of features from tiny images
Krizhevsky, A., Hinton, G., et al · 2009
Earlier work this paper cites.
Reading digits in natural images with unsupervised feature learning
Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., and Ng, A. Y · 2011
Earlier work this paper cites.
Lin, M., Chen, Q., and Yan, S · 2013
Earlier work this paper cites.
Low-rank matrix factorization for deep neural network training with high-dimensional output targets
Sainath, T. N., Kingsbury, B., Sindhwani, V., Arisoy, E., and Ramabhadran, B · 2013
Earlier work this paper cites.
Restructuring of deep neural network acoustic models with singular value decomposition
Xue, J., Li, J., and Gong, Y · 2013
Earlier work this paper cites.
Speeding up convolutional neural networks with low rank expansions
Jaderberg, M., Vedaldi, A., and Zisserman, A · 2014
Earlier work this paper cites.
Very deep convolutional networks for large-scale image recognition
Simonyan, K. and Zisserman, A · 2014
Earlier work this paper cites.
Mean-normalized stochastic gradient for large-scale deep learning
Wiesler, S., Richard, A., Schluter, R., and Ney, H · 2014
Earlier work this paper cites.
Distilling the knowledge in a neural network
Hinton, G., Vinyals, O., and Dean, J · 2015
Earlier work this paper cites.
Training cnns with low-rank filters for efficient image classification
Ioannou, Y., Robertson, D., Shotton, J., Cipolla, R., and Criminisi, A · 2015
Earlier work this paper cites.
Eyeriss: A spatial architecture for energy-efficient dataflow for convolutional neural networks
Chen, Y.-H., Emer, J., and Sze, V · 2016
Earlier work this paper cites.
Deep residual learning for image recognition
He, K., Zhang, X., Ren, S., and Sun, J · 2016
Earlier work this paper cites.
Network trimming: A data-driven neuron pruning approach towards efficient deep architectures
Hu, H., Peng, R., Tai, Y.-W., and Tang, C.-K · 2016
Earlier work this paper cites.
Binarized neural networks
Hubara, I., Courbariaux, M., Soudry, D., El-Yaniv, R., and Bengio, Y · 2016
Earlier work this paper cites.
Squeezenet: Alexnet-level accuracy with 50x fewer parameters and¡ 0.5 mb model size
Iandola, F. N., Han, S., Moskewicz, M. W., Ashraf, K., Dally, W. J., and Keutzer, K · 2016
Earlier work this paper cites.
Intel Xeon Phi processor high performance programming: knights landing edition
Jeffers, J., Reinders, J., and Sodani, A · 2016
Earlier work this paper cites.
Federated learning: Strategies for improving communication efficiency
Konečnỳ, J., McMahan, H. B., Yu, F. X., Richtárik, P., Suresh, A. T., and Bacon, D · 2016
Earlier work this paper cites.
Pruning filters for efficient convnets
Li, H., Kadav, A., Durdanovic, I., Samet, H., and Graf, H. P · 2016
Earlier work this paper cites.
Xnor-net: Imagenet classification using binary convolutional neural networks
Rastegari, M., Ordonez, V., Redmon, J., and Farhadi, A · 2016
Earlier work this paper cites.
Learning structured sparsity in deep neural networks
Wen, W., Wu, C., Wang, Y., Chen, Y., and Li, H · 2016
Earlier work this paper cites.
Quantized convolutional neural networks for mobile devices
Wu, J., Leng, C., Wang, Y., Hu, Q., and Cheng, J · 2016
Earlier work this paper cites.
Zagoruyko, S. and Komodakis, N · 2016
Earlier work this paper cites.
Zhu, C., Han, S., Mao, H., and Dally, W. J · 2016
Cited alongside, same era.
Xception: Deep learning with depthwise separable convolutions
Chollet, F · 2017
Cited alongside, same era.
Accurate, large minibatch sgd: Training imagenet in 1 hour
Goyal, P., Dollár, P., Girshick, R., Noordhuis, P., Wesolowski, L., Kyrola, A., Tulloch, A., Jia, Y., and He, K · 2017
Cited alongside, same era.
Channel pruning for accelerating very deep neural networks
He, Y., Zhang, X., and Sun, J · 2017
Cited alongside, same era.
Mobilenets: Efficient convolutional neural networks for mobile vision applications
Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., and Adam, H · 2017
Cited alongside, same era.
Drawing early-bird tickets: Towards more efficient training of deep networks
You, H., Li, C., Xu, P., Fu, Y., Wang, Y., Chen, X., Baraniuk, R. G., Wang, Z., and Lin, Y · 2019
Later among the works it cites.
Universally slimmable networks and improved training techniques
Yu, J. and Huang, T. S · 2019
Later among the works it cites.
Slimmable neural networks
Yu, J., Yang, L., Xu, N., Yang, J., and Huang, T · 2019
Later among the works it cites.
Language models are few-shot learners
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D · 2020
Later among the works it cites.
Rigging the lottery: Making all tickets winners
Evci, U., Gale, T., Menick, J., Castro, P. S., and Elsen, E · 2020
Later among the works it cites.
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Quantized neural networks: Training neural networks with low precision weights and activations
Hubara, I., Courbariaux, M., Soudry, D., El-Yaniv, R., and Bengio, Y · 2017
Cited alongside, same era.
Attention is all you need
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I · 2017
Cited alongside, same era.
Designing energy-efficient convolutional neural networks using energy-aware pruning
Yang, T.-J., Chen, Y.-H., and Sze, V · 2017
Cited alongside, same era.
Incremental network quantization: Towards lossless cnns with low-precision weights
Zhou, A., Yao, A., Guo, Y., Xu, L., and Chen, Y · 2017
Cited alongside, same era.
To prune, or not to prune: exploring the efficacy of pruning for model compression
Zhu, M. and Gupta, S · 2017
Cited alongside, same era.
Bert: Pre-training of deep bidirectional transformers for language understanding
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K · 2018
Cited alongside, same era.
The lottery ticket hypothesis: Finding sparse, trainable neural networks
Frankle, J. and Carbin, M · 2018
Cited alongside, same era.
Low-rank compression of neural nets: Learning the rank of each layer
Idelbayev, Y. and Carreira-Perpinán, M. A · 2020
Later among the works it cites.
Tinybert: Distilling bert for natural language understanding
Jiao, X., Yin, Y., Shang, L., Jiang, X., Chen, X., Li, L., Wang, F., and Liu, Q · 2020
Later among the works it cites.
Initialization and regularization of factorized neural layers
Khodak, M., Tenenholtz, N. A., Mackey, L., and Fusi, N · 2020
Later among the works it cites.
Reformer: The efficient transformer
Kitaev, N., Kaiser, L., and Levskaya, A · 2020
Later among the works it cites.
Comparing rewinding and fine-tuning in neural network pruning
Renda, A., Frankle, J., and Carbin, M · 2020
Later among the works it cites.
Principal component networks: Parameter reduction early in training
Waleffe, R. and Rekatsinas, T · 2020
Later among the works it cites.
Drawing early-bird tickets: Toward more efficient training of deep networks
You, H., Li, C., Xu, P., Fu, Y., Wang, Y., Chen, X., Baraniuk, R. G., Wang, Z., and Lin, Y · 2020
Later among the works it cites.
Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity
Fedus, W., Zoph, B., and Shazeer, N · 2021
Later among the works it cites.
Lora: Low-rank adaptation of large language models
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W · 2021
Later among the works it cites.
How to train bert with an academic budget
Izsak, P., Berchansky, M., and Levy, O · 2021
Later among the works it cites.
Mlp-mixer: An all-mlp architecture for vision
Tolstikhin, I. O., Houlsby, N., Kolesnikov, A., Beyer, L., Zhai, X., Unterthiner, T., Yung, J., Steiner, A., Keysers, D., Uszkoreit, J., et al · 2021
Later among the works it cites.
Fedhm: Efficient federated learning for heterogeneous models via low-rank factorization
Yao, D., Pan, W., Wan, Y., Jin, H., and Sun, L · 2021
Later among the works it cites.
Pixelated butterfly: Simple and efficient sparse training for neural network models
Chen, B., Dao, T., Liang, K., Yang, J., Song, Z., Rudra, A., and Re, C · 2022
Later among the works it cites.
Monarch: Expressive structured matrices for efficient and accurate training
Dao, T., Chen, B., Sohoni, N., Desai, A., Poli, M., Grogan, J., Liu, A., Rao, A., Rudra, A., and Ré, C · 2022
Later among the works it cites.
Fedpara: Low-rank hadamard product for communication-efficient federated learning
Hyeon-Woo, N., Ye-Bin, M., and Oh, T.-H · 2022
Later among the works it cites.
Algorithms for efficiently learning low-rank neural networks
Vodrahalli, K., Shivanna, R., Sathiamoorthy, M., Jain, S., and Chi, E · 2022
Later among the works it cites.
Opt: Open pre-trained transformer language models
Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al · 2022
Later among the works it cites.
Mpcformer: fast, performant and private transformer inference with mpc
Li, D., Wang, H., Shao, R., Guo, H., Xing, E., and Zhang, H · 2023
Closest in time.