Fetching the paper…

Reading the bibliography…

Single-Net Continual Learning with Progressive Segmented Training (PST) · Around

2019

Single-Net Continual Learning with Progressive Segmented Training (PST)

Du, Xiaocong, Charan, Gouranga, Liu, Frank et al.

Original
PDF
Post on X

Understand

There is an increasing need of continual learning in dynamic systems, such as the self-driving vehicle, the surveillance drone, and the robotic system.

Such a system requires learning from the data stream, training the model to preserve previous information and adapt to a new task, and generating a single-headed vector for future inference.
Different from previous approaches with dynamic structures, this work focuses on a single network and model segmentation to prevent catastrophic forgetting.
Leveraging the redundant capacity of a single network, model parameters for each task are separated into two groups: one important group which is frozen to preserve current knowledge, and secondary group to be saved (not pruned) for a future learning.

Built on

R. Aljundi, M. Lin, B. Goujaud, and Y. Bengio, “Online continual learning with no task boundaries,” arXiv:1903.08671 [cs, stat] , Mar. 2019, arXiv: 1903.08671. [Online]. Available: http://arxiv.org/abs/1903.08671
Original
1903
Earlier work this paper cites.
J. Zhang, J. Zhang, S. Ghosh, D. Li, J. Zhu, H. Zhang, and Y. Wang, “Regularize, Expand and Compress: Multi-task based Lifelong Learning via NonExpansive AutoML,” arXiv:1903.08362 [cs] , Mar. 2019, arXiv: 1903.08362. [Online]. Available: http://arxiv.org/abs/1903.08362
Original
1903
Earlier work this paper cites.
J. Zhang, J. Zhang, S. Ghosh, D. Li, S. Tasci, L. Heck, H. Zhang, and C.-C. J. Kuo, “Class-incremental Learning via Deep Model Consolidation,” arXiv:1903.07864 [cs] , Mar. 2019, arXiv: 1903.07864. [Online]. Available: http://arxiv.org/abs/1903.07864
Original
1903
Earlier work this paper cites.
Y. Wu, Y. Chen, L. Wang, Y. Ye, Z. Liu, Y. Guo, and Y. Fu, “Large scale incremental learning,” CoRR , vol. abs/1905.13260, 2019. [Online]. Available: http://arxiv.org/abs/1905.13260
Original
1905
Earlier work this paper cites.
M. McCloskey and N. J. Cohen, “Catastrophic interference in connectionist networks: The sequential learning problem,” in Psychology of learning and motivation . Elsevier, 1989, vol. 24, pp. 109–165
1989
Earlier work this paper cites.
A. Krizhevsky and G. Hinton, “Learning multiple layers of features from tiny images,” Citeseer, Tech. Rep., 2009
2009
Earlier work this paper cites.
K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” arXiv preprint arXiv:1409.1556 , 2014
Original
2014
Earlier work this paper cites.
Y. Gong, L. Liu, M. Yang, and L. Bourdev, “Compressing deep convolutional networks using vector quantization,” arXiv preprint arXiv:1412.6115 , 2014
Original
2014
Earlier work this paper cites.
S. Han, J. Pool, J. Tran, and W. Dally, “Learning both weights and connections for efficient neural network,” in Advances in Neural Information Processing Systems , 2015, pp. 1135–1143
2015
Earlier work this paper cites.
S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” arXiv preprint arXiv:1502.03167 , 2015
Original
2015
Earlier work this paper cites.
S.-A. Rebuffi, A. Kolesnikov, G. Sperl, and C. H. Lampert, “iCaRL: Incremental Classifier and Representation Learning,” arXiv:1611.07725 [cs, stat] , Nov. 2016, arXiv: 1611.07725. [Online]. Available: http://arxiv.org/abs/1611.07725
Original
2016
Earlier work this paper cites.
A. A. Rusu, N. C. Rabinowitz, G. Desjardins, H. Soyer, J. Kirkpatrick, K. Kavukcuoglu, R. Pascanu, and R. Hadsell, “Progressive Neural Networks,” arXiv:1606.04671 [cs] , Jun. 2016, arXiv: 1606.04671. [Online]. Available: http://arxiv.org/abs/1606.04671
Original
2016
Earlier work this paper cites.
J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska, D. Hassabis, C. Clopath, D. Kumaran, and R. Hadsell, “Overcoming catastrophic forgetting in neural networks,” arXiv:1612.00796 [cs, stat] , Dec. 2016, arXiv: 1612.00796. [Online]. Available: http://arxiv.org/abs/1612.00796
Original
2016
Earlier work this paper cites.

Similar

K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2016, pp. 770–778
2016
Cited alongside, same era.
P. Molchanov, S. Tyree, T. Karras, T. Aila, and J. Kautz, “Pruning convolutional neural networks for resource efficient inference,” arXiv preprint arXiv:1611.06440 , 2016
Original
2016
Cited alongside, same era.
H. Li, A. Kadav, I. Durdanovic, H. Samet, and H. P. Graf, “Pruning filters for efficient convnets,” arXiv preprint arXiv:1608.08710 , 2016
Original
2016
Cited alongside, same era.
J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska et al. , “Overcoming catastrophic forgetting in neural networks,” Proceedings of the national academy of sciences , vol. 114, no. 13, pp. 3521–3526, 2017
2017
Cited alongside, same era.

Then

J. Serrà, D. Surís, M. Miron, and A. Karatzoglou, “Overcoming catastrophic forgetting with hard attention to the task,” arXiv:1801.01423 [cs, stat] , Jan. 2018, arXiv: 1801.01423. [Online]. Available: http://arxiv.org/abs/1801.01423
Original
2018
Later among the works it cites.
R. Aljundi, F. Babiloni, M. Elhoseiny, M. Rohrbach, and T. Tuytelaars, “Memory Aware Synapses: Learning What (not) to Forget,” in Computer Vision – ECCV 2018 , V. Ferrari, M. Hebert, C. Sminchisescu, and Y. Weiss, Eds. Cham: Springer International Publishing, 2018, vol. 11207, pp. 144–161
2018
Later among the works it cites.
Z. Li and D. Hoiem, “Learning without Forgetting,” IEEE Transactions on Pattern Analysis and Machine Intelligence , vol. 40, no. 12, pp. 2935–2947, Dec. 2018. [Online]. Available: https://ieeexplore.ieee.org/document/8107520/
2018
Later among the works it cites.
A. Mallya, D. Davis, and S. Lazebnik, “Piggyback: Adapting a Single Network to Multiple Tasks by Learning to Mask Weights,” in Computer Vision – ECCV 2018 , V. Ferrari, M. Hebert, C. Sminchisescu, and Y. Weiss, Eds. Cham: Springer International Publishing, 2018, vol. 11208, pp. 72–88

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

F. Zenke, B. Poole, and S. Ganguli, “Continual Learning Through Synaptic Intelligence,” arXiv:1703.04200 [cs, q-bio, stat] , Mar. 2017, arXiv: 1703.04200. [Online]. Available: http://arxiv.org/abs/1703.04200

2017

Cited alongside, same era.

S.-W. Lee, J.-H. Kim, J. Jun, J.-W. Ha, and B.-T. Zhang, “Overcoming catastrophic forgetting by incremental moment matching,” in Advances in neural information processing systems , 2017, pp. 4652–4662

2017

Cited alongside, same era.

D. Lopez-Paz and M. Ranzato, “Gradient Episodic Memory for Continual Learning,” in Advances in Neural Information Processing Systems 30 , I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, Eds. Curran Associates, Inc., 2017, pp. 6467–6476. [Online]. Available: http://papers.nips.cc/paper/7225-gradient-episodic-memory-for-continual-learning.pdf

2017

Cited alongside, same era.

J. Yoon, E. Yang, J. Lee, and S. J. Hwang, “Lifelong Learning with Dynamically Expandable Networks,” arXiv:1708.01547 [cs] , Aug. 2017, arXiv: 1708.01547. [Online]. Available: http://arxiv.org/abs/1708.01547

2017

Cited alongside, same era.

C. V. Nguyen, Y. Li, T. D. Bui, and R. E. Turner, “Variational continual learning,” arXiv preprint arXiv:1710.10628 , 2017

2017

Cited alongside, same era.

I. Hubara, M. Courbariaux, D. Soudry, R. El-Yaniv, and Y. Bengio, “Quantized neural networks: Training neural networks with low precision weights and activations,” The Journal of Machine Learning Research , vol. 18, no. 1, pp. 6869–6898, 2017

2017

Cited alongside, same era.

A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer, “Automatic differentiation in pytorch,” 2017

2017

Cited alongside, same era.

J.-H. Luo, J. Wu, and W. Lin, “Thinet: A filter level pruning method for deep neural network compression,” in Proceedings of the IEEE international conference on computer vision , 2017, pp. 5058–5066

2017

Cited alongside, same era.

2018

Later among the works it cites.

A. Mallya and S. Lazebnik, “PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning,” in 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Salt Lake City, UT: IEEE, Jun. 2018, pp. 7765–7773. [Online]. Available: https://ieeexplore.ieee.org/document/8578908/

2018

Later among the works it cites.

F. M. Castro, M. J. Marín-Jiménez, N. Guil, C. Schmid, and K. Alahari, “End-to-end incremental learning,” in Proceedings of the European Conference on Computer Vision (ECCV) , 2018, pp. 233–248

2018

Later among the works it cites.

K. Javed and F. Shafait, “Revisiting distillation and incremental classifier learning,” arXiv preprint arXiv:1807.02802 , 2018

2018

Later among the works it cites.

S. Farquhar and Y. Gal, “Towards robust evaluations of continual learning,” arXiv preprint arXiv:1805.09733 , 2018

2018

Later among the works it cites.

Z. Liu, M. Sun, T. Zhou, G. Huang, and T. Darrell, “Rethinking the value of network pruning,” arXiv preprint arXiv:1810.05270 , 2018

2018

Later among the works it cites.

A. Chaudhry, M. Ranzato, M. Rohrbach, and M. Elhoseiny, “EFFICIENT LIFELONG LEARNING WITH A-GEM,” p. 20, 2019

2019

Closest in time.

X. Du, Z. Li, Y. Ma, and Y. Cao, “Efficient network construction through structural plasticity,” IEEE Journal on Emerging and Selected Topics in Circuits and Systems , vol. 9, no. 3, pp. 453–464, Sep. 2019

2019

Closest in time.

X. Du, G. Krishnan, A. Mohanty, Z. Li, G. Charan, and Y. Cao, “Towards efficient neural networks on-a-chip: Joint hardware-algorithm approaches,” in 2019 China Semiconductor Technology International Conference (CSTIC) , March 2019, pp. 1–5

2019

Closest in time.

S. Kolala Venkataramanaiah, Y. Ma, S. Yin, E. Nurvithadhi, A. Dasu, Y. Cao, and J.-s. Seo, “Automatic compiler based fpga accelerator for cnn training,” in 2019 29th International Conference on Field Programmable Logic and Applications (FPL) . IEEE, 2019

2019

Closest in time.