2022

Lottery Pools: Winning More by Interpolating Tickets without Increasing Training or Inference Cost

Yin, Lu, Liu, Shiwei, Fang, Meng et al.

Understand

Lottery tickets (LTs) is able to discover accurate and sparse subnetworks that could be trained in isolation to match the performance of dense networks.

Ensemble, in parallel, is one of the oldest time-proven tricks in machine learning to improve performance by combining the output of multiple independent models.
However, the benefits of ensemble in the context of LTs will be diluted since ensemble does not directly lead to stronger sparse subnetworks, but leverages their predictions for a better decision.
In this work, we first observe that directly averaging the weights of the adjacent learned subnetworks significantly boosts the performance of LTs.

Built on

Language Models are Few-Shot Learners
Brown, T.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J. D.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; Agarwal, S.; Herbert-Voss, A.; Krueger, G.; Henighan, T.; Child, R.; Ramesh, A.; Ziegler, D.; Wu, J.; Winter, C.; Hesse, C.; Chen, M.; Sigler, E.; Litwin, M.; Gray, S.; Chess, B.; Clark, J.; Berner, C.; McCandlish, S.; Radford, A.; Sutskever, I.; and Amodei, D. 2020 · 1901
Earlier work this paper cites.
The state of sparsity in deep neural networks
Original
Gale, T.; Elsen, E.; and Hooker, S. 2019 · 1902
Earlier work this paper cites.
Playing the lottery with rewards and multiple languages: lottery tickets in rl and nlp
Original
Yu, H.; Edunov, S.; Tian, Y.; and Morcos, A. S. 2019 · 1906
Earlier work this paper cites.
Deep ensembles: A loss landscape perspective
Original
Fort, S.; Hu, H.; and Lakshminarayanan, B. 2019 · 1912
Earlier work this paper cites.
Efficient estimations from a slowly convergent Robbins-Monro process
Ruppert, D. 1988 · 1988
Earlier work this paper cites.
Handwritten digit recognition with a back-propagation network
LeCun, Y.; Boser, B.; Denker, J.; Henderson, D.; Howard, R.; Hubbard, W.; and Jackel, L. 1989 · 1989
Earlier work this paper cites.
Using relevance to reduce network size automatically
Mozer, M. C.; and Smolensky, P. 1989 · 1989
Earlier work this paper cites.
Neural network ensembles
Hansen, L. K.; and Salamon, P. 1990 · 1990
Earlier work this paper cites.
A statistical approach to learning and generalization in layered neural networks
Levin, E.; Tishby, N.; and Solla, S. A. 1990 · 1990
Earlier work this paper cites.
When networks disagree: Ensemble methods for hybrid neural networks
Perrone, M. P.; and Cooper, L. N. 1992 · 1992
Earlier work this paper cites.
Acceleration of stochastic approximation by averaging
Polyak, B. T.; and Juditsky, A. B. 1992 · 1992
Earlier work this paper cites.
Bagging predictors
Breiman, L. 1996 · 1996
Earlier work this paper cites.
Ensemble Methods in Machine Learning
Dietterich, T. G. 2000 · 2000
Earlier work this paper cites.
Comparing rewinding and fine-tuning in neural network pruning
Original
Renda, A.; Frankle, J.; and Carbin, M. 2020 · 2003
Earlier work this paper cites.
An image is worth 16x16 words: Transformers for image recognition at scale
Original
Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; et al. 2020 · 2010
Earlier work this paper cites.
ImageNet Classification with Deep Convolutional Neural Networks
Krizhevsky, A.; Sutskever, I.; and Hinton, G. E. 2012 · 2012
Earlier work this paper cites.
Adam: A method for stochastic optimization
Original
Kingma, D. P.; and Ba, J. 2014 · 2014
Earlier work this paper cites.
Very deep convolutional networks for large-scale image recognition
Original
Simonyan, K.; and Zisserman, A. 2014 · 2014
Earlier work this paper cites.
Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding
Original
Han, S.; Mao, H.; and Dally, W. J. 2015 · 2015
Earlier work this paper cites.
Deep Residual Learning for Image Recognition
He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016 · 2016
Earlier work this paper cites.

Similar

Pruning convolutional neural networks for resource efficient inference
Original
Molchanov, P.; Tyree, S.; Karras, T.; Aila, T.; and Kautz, J. 2016 · 2016
Cited alongside, same era.
Mastering the game of Go with deep neural networks and tree search
Silver, D.; Huang, A.; Maddison, C. J.; Guez, A.; Sifre, L.; Van Den Driessche, G.; Schrittwieser, J.; Antonoglou, I.; Panneershelvam, V.; Lanctot, M.; et al. 2016 · 2016
Cited alongside, same era.
Snapshot ensembles: Train 1, get m for free
Original
Huang, G.; Li, Y.; Pleiss, G.; Liu, Z.; Hopcroft, J. E.; and Weinberger, K. Q. 2017 · 2017
Cited alongside, same era.
Progressive growing of gans for improved quality, stability, and variation
Original
Karras, T.; Aila, T.; Laine, S.; and Lehtinen, J. 2017 · 2017
Cited alongside, same era.
Simple and scalable predictive uncertainty estimation using deep ensembles
Lakshminarayanan, B.; Pritzel, A.; and Blundell, C. 2017 · 2017
Cited alongside, same era.
Decoupled weight decay regularization
Original
Loshchilov, I.; and Hutter, F. 2017 · 2017
Cited alongside, same era.
The lottery ticket hypothesis: Finding sparse, trainable neural networks
Original
Frankle, J.; and Carbin, M. 2018 · 2018
Cited alongside, same era.
Loss surfaces, mode connectivity, and fast ensembling of dnns
Garipov, T.; Izmailov, P.; Podoprikhin, D.; Vetrov, D.; and Wilson, A. G. 2018 · 2018
Cited alongside, same era.
Averaging weights leads to wider optima and better generalization
Original
Izmailov, P.; Podoprikhin, D.; Garipov, T.; Vetrov, D.; and Wilson, A. G. 2018 · 2018
Cited alongside, same era.
Estimation of energy consumption in machine learning
García-Martín, E.; Rodrigues, C. F.; Riley, G.; and Grahn, H. 2019 · 2019
Cited alongside, same era.
One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizers
Morcos, A.; Yu, H.; Paganini, M.; and Tian, Y. 2019 · 2019
Cited alongside, same era.
Uniform convergence may be unable to explain generalization in deep learning
Nagarajan, V.; and Kolter, J. Z. 2019 · 2019
Cited alongside, same era.

Then

What’s hidden in a randomly weighted neural network?
Ramanujan, V.; Wortsman, M.; Kembhavi, A.; Farhadi, A.; and Rastegari, M. 2020 · 2020
Later among the works it cites.
Green ai
Schwartz, R.; Dodge, J.; Smith, N. A.; and Etzioni, O. 2020 · 2020
Later among the works it cites.
BatchEnsemble: an Alternative Approach to Efficient Ensemble and Lifelong Learning
Wen, Y.; Tran, D.; and Ba, J. 2020 · 2020
Later among the works it cites.
Gans can play lottery tickets too
Original
Chen, X.; Zhang, Z.; Sui, Y.; and Chen, T. 2021 · 2021
Later among the works it cites.
Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity
Original
Fedus, W.; Zoph, B.; and Shazeer, N. 2021 · 2021
Later among the works it cites.
Highly accurate protein structure prediction with AlphaFold
Jumper, J.; Evans, R.; Pritzel, A.; Green, T.; Figurnov, M.; Ronneberger, O.; Tunyasuvunakool, K.; Bates, R.; Žídek, A.; Potapenko, A.; et al. 2021 · 2021
Later among the works it cites.
Deep ensembling with no overhead for either training or testing: The all-round blessings of dynamic sparsity
Original
Liu, S.; Chen, T.; Atashgahi, Z.; Chen, X.; Sokar, G.; Mocanu, E.; Pechenizkiy, M.; Wang, Z.; and Mocanu, D. C. 2021 · 2021
Later among the works it cites.
Carbon emissions and large neural network training
Original
Patterson, D.; Gonzalez, J.; Le, Q.; Liang, C.; Munguia, L.-M.; Rothchild, D.; So, D.; Texier, M.; and Dean, J. 2021 · 2021
Later among the works it cites.
Learning transferable visual models from natural language supervision
Original
Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al. 2021 · 2021
Later among the works it cites.
Load-balanced Gather-scatter Patterns for Sparse Deep Neural Networks
Original
Sun, F.; Qin, M.; Zhang, T.; Ma, X.; Li, H.; Luo, J.; Zhao, Z.; Chen, Y.-K.; and Xie, Y. 2021 · 2021
Later among the works it cites.
Mest: Accurate and fast memory-economic sparse training framework on the edge
Yuan, G.; Ma, X.; Niu, W.; Li, Z.; Kong, Z.; Liu, N.; Gong, Y.; Zhan, Z.; He, C.; Jin, Q.; et al. 2021 · 2021
Later among the works it cites.
Coarsening the granularity: Towards structurally sparse lottery tickets
Original
Chen, T.; Chen, X.; Ma, X.; Wang, Y.; and Wang, Z. 2022 · 2022
Closest in time.
Gradient flow in sparse neural networks and how lottery tickets win
Evci, U.; Ioannou, Y.; Keskin, C.; and Dauphin, Y. 2022 · 2022
Closest in time.
Patching open-vocabulary models by interpolating weights
Ilharco, G.; Wortsman, M.; Gadre, S. Y.; Song, S.; Hajishirzi, H.; Kornblith, S.; Farhadi, A.; and Schmidt, L. 2022 · 2022
Closest in time.
The unreasonable effectiveness of random pruning: Return of the most naive baseline for sparse training
Original
Liu, S.; Chen, T.; Chen, X.; Shen, L.; Mocanu, D. C.; Wang, Z.; and Pechenizkiy, M. 2022 · 2022
Closest in time.
Diverse Weight Averaging for Out-of-Distribution Generalization
Original
Rame, A.; Kirchmeyer, M.; Rahier, T.; Rakotomamonjy, A.; Gallinari, P.; and Cord, M. 2022 · 2022
Closest in time.
Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time
Wortsman, M.; Ilharco, G.; Gadre, S. Y.; Roelofs, R.; Gontijo-Lopes, R.; Morcos, A. S.; Namkoong, H.; Farhadi, A.; Carmon, Y.; Kornblith, S.; et al. 2022 · 2022
Closest in time.
Superposing Many Tickets into One: A Performance Booster for Sparse Neural Network Training
Original
Yin, L.; Menkovski, V.; Fang, M.; Huang, T.; Pei, Y.; Pechenizkiy, M.; Mocanu, D. C.; and Liu, S. 2022 · 2022
Closest in time.
A Survey for Efficient Open Domain Question Answering
Original
Zhang, Q.; Chen, S.; Xu, D.; Cao, Q.; Chen, X.; Cohn, T.; and Fang, M. 2022 · 2022
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…