Fetching the paper…
Reading the bibliography…
As the size of large language models continue to scale, so does the computational resources required to run it.
Recherches quantitatives sur l’excitation electrique des nerfs traitee comme une polarization
Louis Lapique · 1907
Earlier work this paper cites.
Single headed attention RNN: stop thinking with your head
Stephen Merity · 1911
Earlier work this paper cites.
A quantitative description of membrane current and its application to conduction and excitation in nerve
Alan L Hodgkin and Andrew F Huxley · 1952
Earlier work this paper cites.
The perceptron: A probabilistic model for information storage and organization in the brain
Frank Rosenblatt · 1958
Earlier work this paper cites.
Long short-term memory
Sepp Hochreiter and Jürgen Schmidhuber · 1997
Earlier work this paper cites.
Networks of spiking neurons: the third generation of neural network models
Wolfgang Maass · 1997
Earlier work this paper cites.
GLU variants improve transformer
Noam Shazeer · 2002
Earlier work this paper cites.
A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts
Bo Pang and Lillian Lee · 2004
Earlier work this paper cites.
Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales
Bo Pang and Lillian Lee · 2005
Earlier work this paper cites.
Lapicque’s 1907 paper: From frogs to integrate-and-fire
Nicolas Brunel and Mark CW Van Rossum · 2007
Earlier work this paper cites.
Large text compression benchmark, 2011
Matt Mahoney · 2011
Earlier work this paper cites.
Generating sequences with recurrent neural networks
Alex Graves · 2013
Earlier work this paper cites.
Recursive deep models for semantic compositionality over a sentiment treebank
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts · 2013
Earlier work this paper cites.
1.1 computing’s energy problem (and what we can do about it)
Mark Horowitz · 2014
Earlier work this paper cites.
Convolutional neural networks for sentence classification
Yoon Kim · 2014
Earlier work this paper cites.
A million spiking-neuron integrated circuit with a scalable communication network and interface
Paul A Merolla, John V Arthur, Rodrigo Alvarez-Icaza, Andrew S Cassidy, Jun Sawada, Filipp Akopyan, Bryan L Jackson, Nabil Imam, Chen Guo, Yutaka Nakamura, et al · 2014
Earlier work this paper cites.
Improved semantic representations from tree-structured long short-term memory networks
Kai Sheng Tai, Richard Socher, and Christopher D Manning · 2015
Earlier work this paper cites.
Conversion of artificial recurrent neural networks to spiking neural networks for low-power neuromorphic hardware
Peter U Diehl, Guido Zarrella, Andrew Cassidy, Bruno U Pedroni, and Emre Neftci · 2016
Earlier work this paper cites.
Language modeling with gated convolutional networks
Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier · 2017
Earlier work this paper cites.
Pointer sentinel mixture models
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher · 2017
Earlier work this paper cites.
Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin · 2017
Earlier work this paper cites.
Loihi: A neuromorphic manycore processor with on-chip learning
Mike Davies, Narayan Srinivasa, Tsung-Han Lin, Gautham Chinya, Yongqiang Cao, Sri Harsha Choday, Georgios Dimou, Prasad Joshi, Nabil Imam, Shweta Jain, et al · 2018
Earlier work this paper cites.
Deep learning with spiking neurons: Opportunities and challenges
Michael Pfeiffer and Thomas Pfeil · 2018
Earlier work this paper cites.
Improving language understanding by generative pre-training, 2018
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever · 2018
Earlier work this paper cites.
Spatio-temporal backpropagation for training high-performance spiking neural networks
Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, and Luping Shi · 2018
Earlier work this paper cites.
Bert: Pre-training of deep bidirectional transformers for language understanding
Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova · 2019
Earlier work this paper cites.
Surrogate gradient learning in spiking neural networks: Bringing the power of gradient-based optimization to spiking neural networks
Emre O Neftci, Hesham Mostafa, and Friedemann Zenke · 2019
Cited alongside, same era.
Pytorch: An imperative style, high-performance deep learning library
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al · 2019
Cited alongside, same era.
Language models are unsupervised multitask learners
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al · 2019
Cited alongside, same era.
Towards spike-based machine intelligence with neuromorphic computing
Kaushik Roy, Akhilesh Jaiswal, and Priyadarshini Panda · 2019
Cited alongside, same era.
Carbontracker: Tracking and predicting the carbon footprint of training deep learning models
Lasse F Wolff Anthony, Benjamin Kanding, and Raghavendra Selvan · 2020
Cited alongside, same era.
Memristor-based binarized spiking neural networks: Challenges and applications
Jason K Eshraghian, Xinxin Wang, and Wei D Lu · 2022
Later among the works it cites.
Spikingjelly
Wei Fang, Yanqi Chen, Jianhao Ding, Ding Chen, Zhaofei Yu, Huihui Zhou, Timothée Masquelier, Yonghong Tian, and other contributors · 2022
Later among the works it cites.
Training compute-optimal large language models
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al · 2022
Later among the works it cites.
Spikeformer: A novel architecture for training high-performance low-latency spiking neural network
Yudong Li, Yunlin Lei, and Xu Yang · 2022
Later among the works it cites.
Intelligence processing units accelerate neuromorphic learning
Pao-Sheng Vincent Sun, Alexander Titterton, Anjlee Gopiani, Tim Santos, Arindam Basu, Wei D Lu, and Jason K Eshraghian · 2022
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Language models are few-shot learners
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al · 2020
Cited alongside, same era.
Rethinking attention with performers
Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al · 2020
Cited alongside, same era.
The carbon impact of artificial intelligence
Payal Dhar · 2020
Cited alongside, same era.
The pile: An 800gb dataset of diverse text for language modeling
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al · 2020
Cited alongside, same era.
Scaling laws for neural language models
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei · 2020
Cited alongside, same era.
Transformers are rnns: Fast autoregressive transformers with linear attention
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret · 2020
Cited alongside, same era.
Spiking-yolo: spiking neural network for energy-efficient object detection
Seijoon Kim, Seongsik Park, Byunggook Na, and Sungroh Yoon · 2020
Cited alongside, same era.
Later among the works it cites.
Outlier suppression: Pushing the limit of low-bit transformer language models
Xiuying Wei, Yunchen Zhang, Xiangguo Zhang, Ruihao Gong, Shanghang Zhang, Qi Zhang, Fengwei Yu, and Xianglong Liu · 2022
Later among the works it cites.
Towards energy-preserving natural language understanding with spiking neural networks
Rong Xiao, Yu Wan, Baosong Yang, Haibo Zhang, Huajin Tang, Derek F Wong, and Boxing Chen · 2022
Later among the works it cites.
Tcja-snn: Temporal-channel joint attention for spiking neural networks
Rui-Jie Zhu, Qihang Zhao, Tianjing Zhang, Haoyu Deng, Yule Duan, Malu Zhang, and Liang-Jian Deng · 2022
Later among the works it cites.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al · 2023
Closest in time.
Spikingbert: Distilling bert to train spiking language models using implicit differentiation
Malyaban Bal and Abhronil Sengupta · 2023
Closest in time.
Spiking neural networks for frame-based and event-based single object localization
Sami Barchid, José Mennesson, Jason Eshraghian, Chaabane Djéraba, and Mohammed Bennamoun · 2023
Closest in time.
The minipile challenge for data-efficient language models
Jean Kaddour · 2023
Closest in time.
Llm-qat: Data-free quantization aware training for large language models
Zechun Liu, Barlas Oguz, Changsheng Zhao, Ernie Chang, Pierre Stock, Yashar Mehdad, Yangyang Shi, Raghuraman Krishnamoorthi, and Vikas Chandra · 2023
Closest in time.
Implementing and benchmarking the locally competitive algorithm on the loihi 2 neuromorphic processor
Gavin Parpart, Sumedh Risbud, Garrett Kenyon, and Yijing Watkins · 2023
Closest in time.
Rwkv: Reinventing rnns for the transformer era
Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, Kranthi Kiran GV, et al · 2023
Closest in time.
Is conventional snn really efficient? a perspective from network quantization
Guobin Shen, Dongcheng Zhao, Tenglong Li, Jindong Li, and Yi Zeng · 2023
Closest in time.
Smoothquant: Accurate and efficient post-training quantization for large language models
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han · 2023
Closest in time.
Baichuan 2: Open large-scale language models
Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, et al · 2023
Closest in time.
Attention spiking neural networks
Man Yao, Guangshe Zhao, Hengyu Zhang, Yifan Hu, Lei Deng, Yonghong Tian, Bo Xu, and Guoqi Li · 2023
Closest in time.
Spikformer: When spiking neural network meets transformer
Zhaokun Zhou, Yuesheng Zhu, Chao He, Yaowei Wang, Shuicheng YAN, Yonghong Tian, and Li Yuan · 2023
Closest in time.
Tensor decomposition based attention module for spiking neural networks
Haoyu Deng, Ruijie Zhu, Xuerui Qiu, Yule Duan, Malu Zhang, and Liang-Jian Deng · 2024
Closest in time.
Optically tunable electrical oscillations in oxide-based memristors for neuromorphic computing
Shimul Kanti Nath, Sujan Kumar Das, Sanjoy Kumar Nandi, Chen Xi, Camilo Verbel Marquez, Armando Rúa, Mutsunori Uenuma, Zhongrui Wang, Songqing Zhang, Rui-Jie Zhu, et al · 2024
Closest in time.
Gated attention coding for training high-performance and efficient spiking neural networks
Xuerui Qiu, Rui-Jie Zhu, Yuhong Chou, Zhaorui Wang, Liang-jian Deng, and Guoqi Li · 2024
Closest in time.
Squat: Stateful quantization-aware training in recurrent spiking neural networks
Sreyes Venkatesh, Razvan Marinescu, and Jason K Eshraghian · 2024
Closest in time.
Scalable matmul-free language modeling
Rui-Jie Zhu, Yu Zhang, Ethan Sifferman, Tyler Sheaves, Yiqiao Wang, Dustin Richmond, Peng Zhou, and Jason K Eshraghian · 2024
Closest in time.