Fetching the paper…
Reading the bibliography…
The past year has witnessed the increasing popularity of Large Language Models (LLMs).
T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei, “Language models are few-shot learners,” in Advances in neural information processing systems , vol. 33, 2020, pp. 1877–1901
1901
Earlier work this paper cites.
M. D. Lam, E. E. Rothberg, and M. E. Wolf, “The cache performance and optimizations of blocked algorithms,” ACM SIGOPS Operating Systems Review , vol. 25, no. Special Issue, pp. 63–74, 1991
1991
Earlier work this paper cites.
A. Alexandrov, M. F. Ionescu, K. E. Schauser, and C. Scheiman, “Loggp: Incorporating long messages into the logp model—one step closer towards a realistic model for parallel computation,” in Proceedings of the seventh annual ACM symposium on Parallel algorithms and architectures , 1995, pp. 95–105
1995
Earlier work this paper cites.
A. Bakhoda, G. L. Yuan, W. W. Fung, H. Wong, and T. M. Aamodt, “Analyzing cuda workloads using a detailed gpu simulator,” in 2009 IEEE international symposium on performance analysis of systems and software . IEEE, 2009, pp. 163–174
2009
Earlier work this paper cites.
N. Muralimanohar, R. Balasubramonian, and N. P. Jouppi, “Cacti 6.0: A tool to model large caches,” HP Laboratories, Tech. Rep. HPL-2009-85, 2009
2009
Earlier work this paper cites.
P. Patarasuk and X. Yuan, “Bandwidth optimal all-reduce algorithms for clusters of workstations,” Journal of Parallel and Distributed Computing , vol. 69, no. 2, pp. 117–124, 2009
2009
Earlier work this paper cites.
P. Raghavan, A. Lambrechts, M. Jayapala, F. Catthoor, and D. Verkest, “Empire: Empirical power/area/timing models for register files,” Microprocessors and Microsystems , vol. 33, no. 4, pp. 295–300, 2009, media and Stream Processing. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0141933109000258
2009
Earlier work this paper cites.
S. Williams, A. Waterman, and D. Patterson, “Roofline: an insightful visual performance model for multicore architectures,” Communications of the ACM , vol. 52, no. 4, pp. 65–76, 2009
2009
Earlier work this paper cites.
H. Kim, J. Lee, N. B. Lakshminarayana, J. Sim, J. Lim, and T. Pho, “Macsim: A cpu-gpu heterogeneous simulation framework user guide,” Georgia Institute of Technology , 2012
2012
Earlier work this paper cites.
R. Ubal, B. Jang, P. Mistry, D. Schaa, and D. Kaeli, “Multi2sim: A simulation framework for cpu-gpu computing,” in Proceedings of the 21st international conference on Parallel architectures and compilation techniques , 2012, pp. 335–344
2012
Earlier work this paper cites.
B. M. Beckmann and A. Gutierrez, “The amd gem5 apu simulator: Modeling heterogeneous systems in gem5,” in Tutorial at the International Symposium on Microarchitecture (MICRO) , 2015
2015
Earlier work this paper cites.
Y.-H. Chen, J. Emer, and V. Sze, “Eyeriss: A spatial architecture for energy-efficient dataflow for convolutional neural networks,” SIGARCH Comput. Archit. News , vol. 44, no. 3, p. 367–379, jun 2016. [Online]. Available: https://doi.org/10.1145/3007787.3001177
2016
Earlier work this paper cites.
2016
Earlier work this paper cites.
D. Foley and J. Danskin, “Ultra-performance pascal gpu and nvlink interconnect,” IEEE Micro , vol. 37, no. 2, pp. 7–17, 2017
2017
Earlier work this paper cites.
X. Gong, R. Ubal, and D. Kaeli, “Multi2sim kepler: A detailed architectural gpu simulator,” in 2017 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS) . IEEE, 2017, pp. 269–278
2017
Earlier work this paper cites.
W. Lu, G. Yan, J. Li, S. Gong, Y. Han, and X. Li, “Flexflow: A flexible dataflow accelerator architecture for convolutional neural networks,” in 2017 IEEE International Symposium on High Performance Computer Architecture (HPCA) . IEEE, 2017, pp. 553–564
2017
Earlier work this paper cites.
B. Reagen, J. M. Hernández-Lobato, R. Adolf, M. Gelbart, P. Whatmough, G.-Y. Wei, and D. Brooks, “A case for efficient accelerator design space exploration via bayesian optimization,” in 2017 IEEE/ACM International Symposium on Low Power Electronics and Design (ISLPED) . IEEE, 2017, pp. 1–6
2017
Earlier work this paper cites.
2018
Earlier work this paper cites.
P. Gera, H. Kim, H. Kim, S. Hong, V. George, and C.-K. Luk, “Performance characterisation and simulation of intel’s integrated gpu architecture,” in 2018 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS) . IEEE, 2018, pp. 139–148
2018
Earlier work this paper cites.
A. Gutierrez, B. M. Beckmann, A. Dutu, J. Gross, M. LeBeane, J. Kalamatianos, O. Kayiran, M. Poremba, B. Potter, S. Puthoor, M. D. Sinclair, M. Wyse, J. Yin, X. Zhang, A. Jain, and T. G. Rogers, “Lost in abstraction: Pitfalls of analyzing gpus at the intermediate language level,” in 2018 IEEE International Symposium on High Performance Computer Architecture (HPCA) . IEEE, 2018, pp. 608–619
2018
Earlier work this paper cites.
M. McKeown, A. Lavrov, M. Shahrad, P. J. Jackson, Y. Fu, J. Balkind, T. M. Nguyen, K. Lim, Y. Zhou, and D. Wentzlaff, “Power and energy characterization of an open source 25-core manycore processor,” in 2018 IEEE International Symposium on High Performance Computer Architecture (HPCA) , 2018, pp. 762–775
2018
Earlier work this paper cites.
2018
Earlier work this paper cites.
NVIDIA Corporation, “The nvidia deep learning accelerator,” NVIDIA, Tech. Rep., 2018. [Online]. Available: https://old.hotchips.org/hc30/2conf/2.08_NVidia_DLA_Nvidia_DLA_HotChips_10Aug18.pdf
2018
Earlier work this paper cites.
2018
Earlier work this paper cites.
H. A. Abdelhafez, C. Zimmer, S. S. Vazhkudai, and M. Ripeanu, “Ahead: A tool for projecting next-generation hardware enhancements on gpu-accelerated systems,” in 2019 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW) , 2019, pp. 583–592
2019
Cited alongside, same era.
S. Dave, Y. Kim, S. Avancha, K. Lee, and A. Shrivastava, “Dmazerunner: Executing perfectly nested loops on dataflow accelerators,” ACM Transactions on Embedded Computing Systems (TECS) , vol. 18, no. 5s, pp. 1–27, 2019
2019
Cited alongside, same era.
M. Lapedus, “What’s next for high bandwidth memory,” Semiconductor Engineering , 2019. [Online]. Available: https://semiengineering.com/whats-next-for-high-bandwidth-memory/
2019
Cited alongside, same era.
A. Parashar, P. Raina, Y. S. Shao, Y.-H. Chen, V. A. Ying, A. Mukkara, R. Venkatesan, B. Khailany, S. W. Keckler, and J. Emer, “Timeloop: A systematic approach to dnn accelerator evaluation,” in 2019 IEEE international symposium on performance analysis of systems and software (ISPASS) . IEEE, 2019, pp. 304–315
H. Wang, Z. Zhang, and S. Han, “Spatten: Efficient sparse attention architecture with cascade token and head pruning,” in 2021 IEEE International Symposium on High-Performance Computer Architecture (HPCA) . IEEE, 2021, pp. 97–110
2021
Later among the works it cites.
R. Y. Aminabadi, S. Rajbhandari, A. A. Awan, C. Li, D. Li, E. Zheng, O. Ruwase, S. Smith, M. Zhang, J. Rasley, and Y. He, “Deepspeed- inference: Enabling efficient inference of transformer models at unprecedented scale,” in SC22: International Conference for High Performance Computing, Networking, Storage and Analysis , 2022, pp. 1–15
2022
Later among the works it cites.
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
2019
Cited alongside, same era.
A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, “Language models are unsupervised multitask learners,” OpenAI blog , vol. 1, no. 8, p. 9, 2019
2019
Cited alongside, same era.
Y. Sun, T. Baruah, S. A. Mojumder, S. Dong, X. Gong, S. Treadway, Y. Bao, S. Hance, C. McCardwell, V. Zhao, H. Barclay, A. K. Ziabari, Z. Chen, R. Ubal, J. L. Abellán, J. Kim, A. Joshi, and D. Kaeli, “Mgpusim: Enabling multi-gpu performance modeling and optimization,” in Proceedings of the 46th International Symposium on Computer Architecture , ser. ISCA ’19. New York, NY, USA: Association for Computing Machinery, 2019, p. 197–209. [Online]. Available: https://doi.org/10.1145/3307650.3322230
2019
Cited alongside, same era.
R. Venkatesan, Y. S. Shao, M. Wang, J. Clemons, S. Dai, M. Fojtik, B. Keller, A. Klinefelter, N. Pinckney, P. Raina, Y. Zhang, B. Zimmer, W. J. Dally, J. Emer, S. W. Keckler, and B. Khailany, “Magnet: A modular accelerator generator for neural networks,” in 2019 IEEE/ACM International Conference on Computer-Aided Design (ICCAD) . IEEE, 2019, pp. 1–8
2019
Cited alongside, same era.
F. Zaruba and L. Benini, “The cost of application-class processing: Energy and performance analysis of a linux-ready 1.7-ghz 64-bit risc-v core in 22-nm fdsoi technology,” IEEE Transactions on Very Large Scale Integration (VLSI) Systems , vol. 27, no. 11, pp. 2629–2640, 2019
2019
Cited alongside, same era.
S. Dave, A. Shrivastava, Y. Kim, S. Avancha, and K. Lee, “dmazerunner: Optimizing convolutions on dataflow accelerators,” in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2020, pp. 1544–1548
2020
Cited alongside, same era.
N. P. Jouppi, D. H. Yoon, G. Kurian, S. Li, N. Patil, J. Laudon, C. Young, and D. Patterson, “A domain-specific supercomputer for training deep neural networks,” Commun. ACM , vol. 63, no. 7, p. 67–78, jun 2020. [Online]. Available: https://doi.org/10.1145/3360307
2020
Cited alongside, same era.
2020
Cited alongside, same era.
M. Khairy, Z. Shen, T. M. Aamodt, and T. G. Rogers, “Accel-sim: An extensible simulation framework for validated gpu modeling,” in 2020 ACM/IEEE 47th Annual International Symposium on Computer Architecture (ISCA) . IEEE, 2020, pp. 473–486
2020
Cited alongside, same era.
2022
Later among the works it cites.
T. Dao, D. Fu, S. Ermon, A. Rudra, and C. Ré, “Flashattention: Fast and memory-efficient exact attention with io-awareness,” Advances in Neural Information Processing Systems , vol. 35, pp. 16 344–16 359, 2022
2022
Later among the works it cites.
W. Fedus, B. Zoph, and N. Shazeer, “Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity,” Journal of Machine Learning Research , vol. 23, no. 120, pp. 1–39, 2022. [Online]. Available: http://jmlr.org/papers/v23/21-0998.html
2022
Later among the works it cites.
J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. van den Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, J. W. Rae, O. Vinyals, and L. Sifre, “Training compute-optimal large language models,” 2022
2022
Later among the works it cites.
2022
Later among the works it cites.
OpenAI. (2022) Introducing chatgpt. [Online]. Available: https://openai.com/blog/chatgpt
2022
Later among the works it cites.
D. Patel, “Nvidia ada lovelace leaked specifications, die sizes, architecture, cost, and performance analysis,” 2022. [Online]. Available: https://www.semianalysis.com/p/nvidia-ada-lovelace-leaked-specifications
2022
Later among the works it cites.
M. Phuong and M. Hutter, “Formal algorithms for transformers,” 2022
2022
Later among the works it cites.
A. Smith and N. James, “Amd instinct™ mi200 series accelerator and node architectures,” in 2022 IEEE Hot Chips 34 Symposium (HCS) , 2022, pp. 1–23
2022
Later among the works it cites.
Y. Wang, Y. Qin, D. Deng, J. Wei, Y. Zhou, Y. Fan, T. Chen, H. Sun, L. Liu, S. Wei, and S. Yin, “A 28nm 27.5tops/w approximate-computing-based transformer processor with asymptotic sparsity speculating and out-of-order computing,” in 2022 IEEE International Solid-State Circuits Conference (ISSCC) , vol. 65, 2022, pp. 1–3
2022
Later among the works it cites.
2022
Later among the works it cites.
D. Zhang, S. Huda, E. Songhori, K. Prabhu, Q. Le, A. Goldie, and A. Mirhoseini, “A full-stack search technique for domain optimized deep learning accelerators,” in Proceedings of the 27th ACM International Conference on Architectural Support for Programming Languages and Operating Systems , 2022, pp. 27–42
2022
Later among the works it cites.
2023
Closest in time.
B. Min, H. Ross, E. Sulem, A. P. B. Veyseh, T. H. Nguyen, O. Sainz, E. Agirre, I. Heintz, and D. Roth, “Recent advances in natural language processing via large pre-trained language models: A survey,” ACM Comput. Surv. , vol. 56, no. 2, sep 2023. [Online]. Available: https://doi.org/10.1145/3605943
2023
Closest in time.
A. Ning, G. Tziantzioulis, and D. Wentzlaff, “Supply chain aware computer architecture,” in Proceedings of the 50th Annual International Symposium on Computer Architecture , ser. ISCA ’23. New York, NY, USA: Association for Computing Machinery, 2023. [Online]. Available: https://doi.org/10.1145/3579371.3589052
2023
Closest in time.
R. Pope, S. Douglas, A. Chowdhery, J. Devlin, J. Bradbury, J. Heek, K. Xiao, S. Agrawal, and J. Dean, “Efficiently scaling transformer inference,” Proceedings of Machine Learning and Systems , vol. 5, 2023
2023
Closest in time.
Y. Sheng, L. Zheng, B. Yuan, Z. Li, M. Ryabinin, B. Chen, P. Liang, C. Ré, I. Stoica, and C. Zhang, “Flexgen: High-throughput generative inference of large language models with a single gpu,” in Proceedings of the 40th International Conference on Machine Learning , ser. ICML’23. JMLR.org, 2023
2023
Closest in time.
B. Tine, V. Saxena, S. Srivatsan, J. R. Simpson, F. Alzammar, L. Cooper, and H. Kim, “Skybox: Open-source graphic rendering on programmable risc-v gpus,” in Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3 , ser. ASPLOS 2023. New York, NY, USA: Association for Computing Machinery, 2023, p. 616–630. [Online]. Available: https://doi.org/10.1145/3582016.3582024
2023
Closest in time.
H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample, “Llama: Open and efficient foundation language models,” 2023
2023
Closest in time.
TrendForce, “Dram spot price,” 2023. [Online]. Available: https://www.dramexchange.com/
2023
Closest in time.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” 2023
2023
Closest in time.