Fetching the paper…

Reading the bibliography…

2022

Neural Architecture Search for Spiking Neural Networks

Kim, Youngeun, Li, Yuhang, Park, Hyoungseob et al.

Original
PDF
Post on X

Understand

Spiking Neural Networks (SNNs) have gained huge attention as a potential energy-efficient alternative to conventional Artificial Neural Networks (ANNs) due to their inherent high-sparsity activation.

However, most prior SNN methods use ANN-like architectures (e.g., VGG-Net or ResNet), which could provide sub-optimal performance for temporal sequence processing of binary information in SNNs.
To address this, in this paper, we introduce a novel Neural Architecture Search (NAS) approach for finding better SNN architectures.
Inspired by recent NAS approaches that find the optimal architecture from activation patterns at initialization, we select the architecture that can represent diverse spike activation patterns across different data samples without training.

Neural Architecture Search for Spiking Neural Networks · Around

Built on

Izhikevich, E.M.: Simple model of spiking neurons. IEEE Transactions on neural networks 14
2003
Earlier work this paper cites.
Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognition. pp. 248–255. Ieee (2009)
2009
Earlier work this paper cites.
Krizhevsky, A., Hinton, G., et al.: Learning multiple layers of features from tiny images (2009)
2009
Earlier work this paper cites.
Jin, X., Rast, A., Galluppi, F., Davies, S., Furber, S.: Implementing spike-timing-dependent plasticity on spinnaker neuromorphic hardware. In: The 2010 International Joint Conference on Neural Networks (IJCNN). pp. 1–8. IEEE (2010)
2010
Earlier work this paper cites.
Montúfar, G., Pascanu, R., Cho, K., Bengio, Y.: On the number of linear regions of deep neural networks. arXiv preprint arXiv:1402.1869 (2014)
Original
2014
Earlier work this paper cites.
Cao, Y., Chen, Y., Khosla, D.: Spiking deep convolutional neural networks for energy-efficient object recognition. International Journal of Computer Vision 113
2015
Earlier work this paper cites.
Diehl, P.U., Cook, M.: Unsupervised learning of digit recognition using spike-timing-dependent plasticity. Frontiers in computational neuroscience 9
2015
Earlier work this paper cites.
Diehl, P.U., Neil, D., Binas, J., Cook, M., Liu, S.C., Pfeiffer, M.: Fast-classifying, high-accuracy spiking deep networks through weight and threshold balancing. In: 2015 International Joint Conference on Neural Networks (IJCNN). pp. 1–8. ieee (2015)
2015
Earlier work this paper cites.
He, K., Zhang, X., Ren, S., Sun, J.: Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In: Proceedings of the IEEE international conference on computer vision. pp. 1026–1034 (2015)
2015
Earlier work this paper cites.
Ioffe, S., Szegedy, C.: Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167 (2015)
Original
2015
Earlier work this paper cites.
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. ICLR (2015)
2015
Earlier work this paper cites.
Baker, B., Gupta, O., Naik, N., Raskar, R.: Designing neural network architectures using reinforcement learning. arXiv preprint arXiv:1611.02167 (2016)
Original
2016
Earlier work this paper cites.
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR. pp. 770–778 (2016)
2016
Earlier work this paper cites.
Lee, J.H., Delbruck, T., Pfeiffer, M.: Training deep spiking neural networks using backpropagation. Frontiers in neuroscience 10
2016
Earlier work this paper cites.
Loshchilov, I., Hutter, F.: Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983 (2016)
Original
2016
Earlier work this paper cites.
Zoph, B., Le, Q.V.: Neural architecture search with reinforcement learning. arXiv preprint arXiv:1611.01578 (2016)
Original
2016
Earlier work this paper cites.
Brock, A., Lim, T., Ritchie, J.M., Weston, N.: Smash: one-shot model architecture search through hypernetworks. arXiv preprint arXiv:1708.05344 (2017)
Original
2017
Earlier work this paper cites.
Mostafa, H.: Supervised learning based on temporal coding in spiking neural networks. IEEE transactions on neural networks and learning systems 29
2017
Earlier work this paper cites.
Panda, P., Roy, K.: Learning to generate sequences with combination of hebbian and non-hebbian plasticity in recurrent spiking neural networks. Frontiers in neuroscience 11
2017
Earlier work this paper cites.
Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., Lerer, A.: Automatic differentiation in pytorch. In: NIPS-W (2017)
2017
Earlier work this paper cites.
Raghu, M., Poole, B., Kleinberg, J., Ganguli, S., Sohl-Dickstein, J.: On the expressive power of deep neural networks. In: international conference on machine learning. pp. 2847–2854. PMLR (2017)
2017
Earlier work this paper cites.
Rueckauer, B., Lungu, I.A., Hu, Y., Pfeiffer, M., Liu, S.C.: Conversion of continuous-valued deep networks to efficient event-driven networks for image classification. Frontiers in neuroscience 11
2017
Earlier work this paper cites.
Bender, G., Kindermans, P.J., Zoph, B., Vasudevan, V., Le, Q.: Understanding and simplifying one-shot architecture search. In: International Conference on Machine Learning. pp. 550–559. PMLR (2018)
2018
Earlier work this paper cites.
Cai, H., Zhu, L., Han, S.: Proxylessnas: Direct neural architecture search on target task and hardware. arXiv preprint arXiv:1812.00332 (2018)
Original
2018
Earlier work this paper cites.
Demin, V., Nekhaev, D.: Recurrent spiking neural network learning based on a competitive maximization of neuronal activity. Frontiers in neuroinformatics 12
2018
Earlier work this paper cites.
Liu, H., Simonyan, K., Yang, Y.: Darts: Differentiable architecture search. arXiv preprint arXiv:1806.09055 (2018)
Original
2018
Earlier work this paper cites.
Pham, H., Guan, M., Zoph, B., Le, Q., Dean, J.: Efficient neural architecture search via parameters sharing. In: International Conference on Machine Learning. pp. 4095–4104. PMLR (2018)
2018
Earlier work this paper cites.
Shrestha, S.B., Orchard, G.: Slayer: Spike layer error reassignment in time. arXiv preprint arXiv:1810.08646 (2018)
Original
2018
Earlier work this paper cites.
Wu, Y., Deng, L., Li, G., Zhu, J., Shi, L.: Spatio-temporal backpropagation for training high-performance spiking neural networks. Frontiers in neuroscience 12
2018
Earlier work this paper cites.
Xie, S., Zheng, H., Liu, C., Lin, L.: Snas: stochastic neural architecture search. arXiv preprint arXiv:1812.09926 (2018)
Original
2018
Earlier work this paper cites.
Yousefzadeh, A., Stromatias, E., Soto, M., Serrano-Gotarredona, T., Linares-Barranco, B.: On practical issues for stochastic stdp hardware with 1-bit synaptic weights. Frontiers in neuroscience 12
2018
Earlier work this paper cites.
Zhong, Z., Yan, J., Wu, W., Shao, J., Liu, C.L.: Practical block-wise neural network architecture generation. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 2423–2432 (2018)
2018
Earlier work this paper cites.
Zoph, B., Vasudevan, V., Shlens, J., Le, Q.V.: Learning transferable architectures for scalable image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 8697–8710 (2018)
2018
Earlier work this paper cites.
Chen, Y., Yang, T., Zhang, X., Meng, G., Xiao, X., Sun, J.: Detnas: Backbone search for object detection. Advances in Neural Information Processing Systems 32
2019
Earlier work this paper cites.
Gong, X., Chang, S., Jiang, Y., Wang, Z.: Autogan: Neural architecture search for generative adversarial networks. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 3224–3234 (2019)
2019
Earlier work this paper cites.
Gu, P., Xiao, R., Pan, G., Tang, H.: Stca: Spatio-temporal credit assignment with delayed feedback in deep spiking neural networks. In: IJCAI. pp. 1366–1372 (2019)
2019
Earlier work this paper cites.

Similar

Hanin, B., Rolnick, D.: Complexity of linear regions in deep networks. In: International Conference on Machine Learning. pp. 2596–2604. PMLR (2019)
2019
Cited alongside, same era.
Hanin, B., Rolnick, D.: Deep relu networks have surprisingly few activation patterns (2019)
2019
Cited alongside, same era.
Liu, C., Chen, L.C., Schroff, F., Adam, H., Hua, W., Yuille, A.L., Fei-Fei, L.: Auto-deeplab: Hierarchical neural architecture search for semantic image segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 82–92 (2019)
2019
Cited alongside, same era.
Neftci, E.O., Mostafa, H., Zenke, F.: Surrogate gradient learning in spiking neural networks. IEEE Signal Processing Magazine 36
2019
Cited alongside, same era.
Real, E., Aggarwal, A., Huang, Y., Le, Q.V.: Regularized evolution for image classifier architecture search. In: Proceedings of the aaai conference on artificial intelligence. vol. 33, pp. 4780–4789 (2019)

Then

Chen, B., Li, P., Li, C., Li, B., Bai, L., Lin, C., Sun, M., Yan, J., Ouyang, W.: Glit: Neural architecture search for global and local image transformer. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 12–21 (2021)
2021
Later among the works it cites.
Chen, W., Gong, X., Wang, Z.: Neural architecture search on imagenet in four gpu hours: A theoretically inspired perspective. arXiv preprint arXiv:2102.11535 (2021)
Original
2021
Later among the works it cites.
Deng, S., Gu, S.: Optimal conversion of conventional artificial neural networks to spiking neural networks. arXiv preprint arXiv:2103.00476 (2021)
Original
2021
Later among the works it cites.
Duan, Y., Chen, X., Xu, H., Chen, Z., Liang, X., Zhang, T., Li, Z.: Transnas-bench-101: Improving transferability and generalizability of cross-task neural architecture search. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 5251–5260 (2021)

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

2019

Cited alongside, same era.

Roy, K., Jaiswal, A., Panda, P.: Towards spike-based machine intelligence with neuromorphic computing. Nature 575

2019

Cited alongside, same era.

Sengupta, A., Ye, Y., Wang, R., Liu, C., Roy, K.: Going deeper in spiking neural networks: Vgg and residual architectures. Frontiers in neuroscience 13

2019

Cited alongside, same era.

Shu, Y., Wang, W., Cai, S.: Understanding architectures learnt by cell-based neural architecture search. arXiv preprint arXiv:1909.09569 (2019)

2019

Cited alongside, same era.

Tan, M., Chen, B., Pang, R., Vasudevan, V., Sandler, M., Howard, A., Le, Q.V.: Mnasnet: Platform-aware neural architecture search for mobile. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2820–2828 (2019)

2019

Cited alongside, same era.

Wu, B., Dai, X., Zhang, P., Wang, Y., Sun, F., Wu, Y., Tian, Y., Vajda, P., Jia, Y., Keutzer, K.: Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10734–10742 (2019)

2019

Cited alongside, same era.

Wu, J., Chua, Y., Zhang, M., Li, G., Li, H., Tan, K.C.: A tandem learning rule for effective training and rapid inference of deep spiking neural networks. arXiv e-prints pp. arXiv–1907 (2019)

2019

Cited alongside, same era.

Wu, Y., Deng, L., Li, G., Zhu, J., Xie, Y., Shi, L.: Direct training for spiking neural networks: Faster, larger, better. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 33, pp. 1311–1318 (2019)

2019

Cited alongside, same era.

2021

Later among the works it cites.

Fang, W., Yu, Z., Chen, Y., Huang, T., Masquelier, T., Tian, Y.: Deep residual learning in spiking neural networks. arXiv preprint arXiv:2102.04159 (2021)

2021

Later among the works it cites.

Fang, W., Yu, Z., Chen, Y., Masquelier, T., Huang, T., Tian, Y.: Incorporating learnable membrane time constant to enhance learning of spiking neural networks. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 2661–2671 (2021)

2021

Later among the works it cites.

Garg, I., Chowdhury, S.S., Roy, K.: Dct-snn: Using dct to distribute spatial information over time for low-latency spiking neural networks. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 4671–4680 (2021)

2021

Later among the works it cites.

Jia, S., Zhang, T., Cheng, X., Liu, H., Xu, B.: Neuronal-plasticity and reward-propagation improved recurrent spiking neural networks. Frontiers in Neuroscience 15

2021

Later among the works it cites.

Kim, Y., Panda, P.: Optimizing deeper spiking neural networks for dynamic vision sensing. Neural Networks (2021)

2021

Later among the works it cites.

Kim, Y., Panda, P.: Visual explanations from spiking neural networks using interspike intervals. Sci Rep 11, 19037 (2021). https://doi.org/10.1038/s41598-021-98448-0 (2021)

2021

Later among the works it cites.

Kim, Y., Venkatesha, Y., Panda, P.: Privatesnn: Fully privacy-preserving spiking neural networks. arXiv preprint arXiv:2104.03414 (2021)

Original

2021

Later among the works it cites.

Kundu, S., Datta, G., Pedram, M., Beerel, P.A.: Spike-thrift: Towards energy-efficient deep spiking neural networks by limiting spiking activity via attention-guided compression. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pp. 3953–3962 (2021)

2021

Later among the works it cites.

Kundu, S., Pedram, M., Beerel, P.A.: Hire-snn: Harnessing the inherent robustness of energy-efficient deep spiking neural networks by training with crafted input noise. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 5209–5218 (2021)

2021

Later among the works it cites.

Li, Y., Deng, S., Dong, X., Gong, R., Gu, S.: A free lunch from ann: Towards efficient, accurate spiking neural networks calibration. arXiv preprint arXiv:2106.06984 (2021)

2021

Later among the works it cites.

Li, Y., Guo, Y., Zhang, S., Deng, S., Hai, Y., Gu, S.: Differentiable spike: Rethinking gradient-descent for training spiking neural networks. Advances in Neural Information Processing Systems 34

2021

Later among the works it cites.

Liang, L., Qu, Z., Chen, Z., Tu, F., Wu, Y., Deng, L., Li, G., Li, P., Xie, Y.: H2learn: High-efficiency learning accelerator for high-accuracy spiking neural networks. arXiv preprint arXiv:2107.11746 (2021)

2021

Later among the works it cites.

Mellor, J., Turner, J., Storkey, A., Crowley, E.J.: Neural architecture search without training. In: International Conference on Machine Learning. pp. 7588–7598. PMLR (2021)

2021

Later among the works it cites.

Rathi, N., Roy, K.: Diet-snn: A low-latency spiking neural network with direct input encoding and leakage and threshold optimization. IEEE Transactions on Neural Networks and Learning Systems (2021)

2021

Later among the works it cites.

Venkatesha, Y., Kim, Y., Tassiulas, L., Panda, P.: Federated learning with spiking neural networks. arXiv preprint arXiv:2106.06579 (2021)

2021

Later among the works it cites.

Wu, H., Zhang, Y., Weng, W., Zhang, Y., Xiong, Z., Zha, Z.J., Sun, X., Wu, F.: Training spiking neural networks with accumulated spiking flow. ijo 1

2021

Later among the works it cites.

Xu, J., Zhao, L., Lin, J., Gao, R., Sun, X., Yang, H.: Knas: green neural architecture search. In: International Conference on Machine Learning. pp. 11613–11625. PMLR (2021)

2021

Later among the works it cites.

Xu, L., Guan, Y., Jin, S., Liu, W., Qian, C., Luo, P., Ouyang, W., Wang, X.: Vipnas: Efficient video pose estimation via neural architecture search. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 16072–16081 (2021)

2021

Later among the works it cites.

Yan, Z., Dai, X., Zhang, P., Tian, Y., Wu, B., Feiszli, M.: Fp-nas: Fast probabilistic neural architecture search. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 15139–15148 (2021)

2021

Later among the works it cites.

Yang, T.J., Liao, Y.L., Sze, V.: Netadaptv2: Efficient neural architecture search with fast super-network training and architecture optimization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2402–2411 (2021)

2021

Later among the works it cites.

Yang, Y., You, S., Li, H., Wang, F., Qian, C., Lin, Z.: Towards improving the consistency, efficiency, and flexibility of differentiable neural architecture search. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 6667–6676 (2021)

2021

Later among the works it cites.

Yang, Z., Wang, Y., Chen, X., Guo, J., Zhang, W., Xu, C., Xu, C., Tao, D., Xu, C.: Hournas: Extremely fast neural architecture search through an hourglass lens. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10896–10906 (2021)

2021

Later among the works it cites.

Yao, M., Gao, H., Zhao, G., Wang, D., Lin, Y., Yang, Z., Li, G.: Temporal-wise attention spiking neural networks for event streams classification. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 10221–10230 (2021)

2021

Later among the works it cites.

Zeng, D., Huang, Y., Bao, Q., Zhang, J., Su, C., Liu, W.: Neural architecture search for joint human parsing and pose estimation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 11385–11394 (2021)

2021

Later among the works it cites.

Zhang, X., Xu, H., Mo, H., Tan, J., Yang, C., Wang, L., Ren, W.: Dcnas: Densely connected neural architecture search for semantic image segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 13956–13967 (2021)

2021

Later among the works it cites.

Zhang, X., Hou, P., Zhang, X., Sun, J.: Neural architecture search with random labels. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10907–10916 (2021)

2021

Later among the works it cites.

Zhao, Y., Wang, L., Tian, Y., Fonseca, R., Guo, T.: Few-shot neural architecture search. In: International Conference on Machine Learning. pp. 12707–12718. PMLR (2021)

2021

Later among the works it cites.

Christensen, D.V., Dittmann, R., Linares-Barranco, B., Sebastian, A., Le Gallo, M., Redaelli, A., Slesazeck, S., Mikolajick, T., Spiga, S., Menzel, S., et al.: 2022 roadmap on neuromorphic computing and engineering. Neuromorphic Computing and Engineering (2022)

2022

Closest in time.

Deng, S., Li, Y., Zhang, S., Gu, S.: Temporal efficient training of spiking neural network via gradient re-weighting. arXiv preprint arXiv:2202.11946 (2022)

2022

Closest in time.

Li, Y., Deng, S., Dong, X., Gu, S.: Converting artificial neural networks to spiking neural networks via parameter calibration. arXiv preprint arXiv:2205.10121 (2022)

2022

Closest in time.

Na, B., Mok, J., Park, S., Lee, D., Choe, H., Yoon, S.: Autosnn: Towards energy-efficient spiking neural networks. arXiv preprint arXiv:2201.12738 (2022)

2022

Closest in time.