Fetching the paper…
Reading the bibliography…
Spiking neural networks (SNNs) have made great progress on both performance and efficiency over the last few years,but their unique working pattern makes it hard to train a high-performance low-latency SNN.Thus the development of SNNs still lags behind traditional artificial neural networks (ANNs).To compensate this gap,many extraordinary works have been proposed.Nevertheless,these works are mainly based on the same kind of network structure (i.e.CNN) and their performance is worse than their ANN counterparts,which limits the applications of SNNs.To this end,we propose a novel Transformer-based SNN,termed "Spikeformer",which outperforms its ANN counterpart on both static dataset and neuromorphic dataset and may be an alternative architecture to CNN for training high-performance SNNs.First,to deal with the problem of "data hungry" and the unstable training period exhibited in the vanilla model,we design the Convolutional Tokenizer (CT) module,which improves the accuracy of the original model on DVS-Gesture by more than 16%.Besides,in order to better incorporate the attention mechanism inside Transformer and the spatio-temporal information inherent to SNN,we adopt spatio-temporal attention (STA) instead of spatial-wise or temporal-wise attention.With our proposed method,we achieve competitive or state-of-the-art (SOTA) SNN performance on DVS-CIFAR10,DVS-Gesture,and ImageNet datasets with the least simulation time steps (i.e.low latency).Remarkably,our Spikeformer outperforms other SNNs on ImageNet by a large margin (i.e.more than 5%) and even outperforms its ANN counterpart by 3.1% and 2.2% on DVS-Gesture and ImageNet respectively,indicating that Spikeformer is a promising architecture for training large-scale SNNs and may be more suitable for SNNs compared to CNN.We believe that this work shall keep the development of SNNs in step with ANNs as much as possible.Code will be available.
Networks of spiking neurons: the third generation of neural network models
Maass, W. 1997 · 1997
Earlier work this paper cites.
A review of the integrate-and-fire neuron model: I. Homogeneous synaptic input
Burkitt, A. N. 2006 · 2006
Earlier work this paper cites.
Imagenet: A large-scale hierarchical image database
Deng, J.; Dong, W.; Socher, R.; Li, L.-J.; Li, K.; and Fei-Fei, L. 2009 · 2009
Earlier work this paper cites.
Adam: A Method for Stochastic Optimization
Kingma, D. P.; and Ba, J. 2015 · 2015
Earlier work this paper cites.
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K.; and Zisserman, A. 2015 · 2015
Earlier work this paper cites.
Ba, J. L.; Kiros, J. R.; and Hinton, G. E. 2016 · 2016
Earlier work this paper cites.
Deep residual learning for image recognition
He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016 · 2016
Earlier work this paper cites.
Deep networks with stochastic depth
Huang, G.; Sun, Y.; Liu, Z.; Sedra, D.; and Weinberger, K. Q. 2016 · 2016
Earlier work this paper cites.
Rethinking the inception architecture for computer vision
Szegedy, C.; Vanhoucke, V.; Ioffe, S.; Shlens, J.; and Wojna, Z. 2016 · 2016
Earlier work this paper cites.
Efficient training of supervised spiking neural network via accurate synaptic-efficiency adjustment method
Xie, X.; Qu, H.; Yi, Z.; and Kurths, J. 2016 · 2016
Earlier work this paper cites.
A Low Power, Fully Event-Based Gesture Recognition System
Amir, A.; Taba, B.; Berg, D. J.; Melano, T.; McKinstry, J. L.; di Nolfo, C.; Nayak, T. K.; Andreopoulos, A.; Garreau, G.; Mendoza, M.; Kusnitz, J. A.; DeBole, M. V.; Esser, S. K.; Delbrück, T.; Flickner, M.; and Modha, D. S. 2017 · 2017
Earlier work this paper cites.
Cifar10-dvs: an event-stream dataset for object classification
Li, H.; Liu, H.; Ji, X.; Li, G.; and Shi, L. 2017 · 2017
Earlier work this paper cites.
SGDR: Stochastic Gradient Descent with Warm Restarts
Loshchilov, I.; and Hutter, F. 2017 · 2017
Earlier work this paper cites.
Attention is all you need
Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, Ł.; and Polosukhin, I. 2017 · 2017
Earlier work this paper cites.
Loihi: A neuromorphic manycore processor with on-chip learning
Davies, M.; Srinivasa, N.; Lin, T.-H.; Chinya, G.; Cao, Y.; Choday, S. H.; Dimou, G.; Joshi, P.; Imam, N.; Jain, S.; et al. 2018 · 2018
Earlier work this paper cites.
Spiking Deep Residual Networks
Hu, Y.; Tang, H.; and Pan, G. 2018 · 2018
Earlier work this paper cites.
Mixed Precision Training
Micikevicius, P.; Narang, S.; Alben, J.; Diamos, G.; Elsen, E.; Garcia, D.; Ginsburg, B.; Houston, M.; Kuchaiev, O.; Venkatesh, G.; et al. 2018 · 2018
Cited alongside, same era.
Context encoding for semantic segmentation
Zhang, H.; Dana, K.; Shi, J.; Zhang, Z.; Wang, X.; Tyagi, A.; and Agrawal, A. 2018 · 2018
Cited alongside, same era.
Attention mechanisms for object recognition with event-based cameras
Cannici, M.; Ciccone, M.; Romanoni, A.; and Matteucci, M. 2019 · 2019
Cited alongside, same era.
AutoAugment: Learning Augmentation Strategies From Data
Cubuk, E. D.; Zoph, B.; Mane, D.; Vasudevan, V.; and Le, Q. V. 2019 · 2019
Cited alongside, same era.
TrueNorth: Accelerating from zero to 64 million neurons in 10 years
DeBole, M. V.; Taba, B.; Amir, A.; Akopyan, F.; Andreopoulos, A.; Risk, W. P.; Kusnitz, J.; Otero, C. O.; Nayak, T. K.; Appuswamy, R.; et al. 2019 · 2019
Cited alongside, same era.
Bag of tricks for image classification with convolutional neural networks
Vision transformer for small-size datasets
Lee, S. H.; Lee, S.; and Song, B. C. 2021 · 2021
Later among the works it cites.
Differentiable spike: Rethinking gradient-descent for training spiking neural networks
Li, Y.; Guo, Y.; Zhang, S.; Deng, S.; Hai, Y.; and Gu, S. 2021 · 2021
Later among the works it cites.
Efficient training of visual transformers with small datasets
Liu, Y.; Sangineto, E.; Bi, W.; Sebe, N.; Lepri, B.; and Nadai, M. 2021 · 2021
Later among the works it cites.
MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformer
Mehta, S.; and Rastegari, M. 2021 · 2021
Later among the works it cites.
Spiking Transformer Networks: A Rate Coded Approach for Processing Sequential Data
Mueller, E.; Studenyak, V.; Auge, D.; and Knoll, A. 2021 · 2021
Later among the works it cites.
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
He, T.; Zhang, Z.; Zhang, H.; Zhang, Z.; Xie, J.; and Li, M. 2019 · 2019
Cited alongside, same era.
Surrogate gradient learning in spiking neural networks: Bringing the power of gradient-based optimization to spiking neural networks
Neftci, E. O.; Mostafa, H.; and Zenke, F. 2019 · 2019
Cited alongside, same era.
Pytorch: An imperative style, high-performance deep learning library
Paszke, A.; Gross, S.; Massa, F.; Lerer, A.; Bradbury, J.; Chanan, G.; Killeen, T.; Lin, Z.; Gimelshein, N.; Antiga, L.; et al. 2019 · 2019
Cited alongside, same era.
Going deeper in spiking neural networks: VGG and residual architectures
Sengupta, A.; Ye, Y.; Wang, R.; Liu, C.; and Roy, K. 2019 · 2019
Cited alongside, same era.
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; et al. 2020 · 2020
Cited alongside, same era.
SpikingJelly
Fang, W.; Chen, Y.; Ding, J.; Chen, D.; Yu, Z.; Zhou, H.; Tian, Y.; and other contributors. 2020 · 2020
Cited alongside, same era.
Rmp-snn: Residual membrane potential neuron for enabling deeper high-accuracy and low-latency spiking neural network
Han, B.; Srinivasan, G.; and Roy, K. 2020 · 2020
Cited alongside, same era.
Liaf-net: Leaky integrate and analog fire network for lightweight and efficient spatiotemporal information processing
Wu, Z.; Zhang, H.; Lin, Y.; Li, G.; Wang, M.; and Tang, Y. 2021 · 2021
Later among the works it cites.
Early convolutions help transformers see better
Xiao, T.; Singh, M.; Mintun, E.; Darrell, T.; Dollár, P.; and Girshick, R. 2021 · 2021
Later among the works it cites.
Temporal-wise attention spiking neural networks for event streams classification
Yao, M.; Gao, H.; Zhao, G.; Wang, D.; Lin, Y.; Yang, Z.; and Li, G. 2021 · 2021
Later among the works it cites.
The remarkable robustness of surrogate gradient learning for instilling complex function in spiking neural networks
Zenke, F.; and Vogels, T. P. 2021 · 2021
Later among the works it cites.
Going deeper with directly-trained larger spiking neural networks
Zheng, H.; Wu, Y.; Deng, L.; Hu, Y.; and Li, G. 2021 · 2021
Later among the works it cites.
Better plain ViT baselines for ImageNet-1k
Beyer, L.; Zhai, X.; and Kolesnikov, A. 2022 · 2022
Closest in time.
Neuromorphic Data Augmentation for Training Spiking Neural Networks
Li, Y.; Kim, Y.; Park, H.; Geller, T.; and Panda, P. 2022 · 2022
Closest in time.
Training High-Performance Low-Latency Spiking Neural Networks by Differentiation on Spike Representation
Meng, Q.; Xiao, M.; Yan, S.; Wang, Y.; Lin, Z.; and Luo, Z.-Q. 2022 · 2022
Closest in time.
EventMix: An Efficient Augmentation Strategy for Event-Based Data
Shen, G.; Zhao, D.; and Zeng, Y. 2022 · 2022
Closest in time.
Training vision transformers with only 2040 images
Cao, Y.-H.; Yu, H.; and Wu, J. 2022 · 2040
Closest in time.