Fetching the paper…
Reading the bibliography…
A number of production deep learning clusters have attempted to explore inference hardware for DNN training, at the off-peak serving hours with many inference GPUs idling.
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in 2009 IEEE Conference on Computer Vision and Pattern Recognition , 2009, pp. 248–255
2009
Earlier work this paper cites.
S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” in Proceedings of the 32nd International Conference on International Conference on Machine Learning - Volume 37 , 2015
2015
Earlier work this paper cites.
K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” in International Conference on Learning Representations , 2015
2015
Earlier work this paper cites.
K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2016, pp. 770–778
2016
Earlier work this paper cites.
2016
Earlier work this paper cites.
2017
Earlier work this paper cites.
2017
Earlier work this paper cites.
H. Zhang, K. Dana, J. Shi, Z. Zhang, X. Wang, A. Tyagi, and A. Agrawal, “Context encoding for semantic segmentation,” in 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2018
2018
Earlier work this paper cites.
NVIDIA, “Nvidia-v100-datasheet,” 2018. [Online]. Available: https://images.nvidia.com/content/technologies/volta/pdf/volta-v100-datasheet-update-us-1165301-r5.pdf
2018
Earlier work this paper cites.
N. Wang, J. Choi, D. Brand, C.-Y. Chen, and K. Gopalakrishnan, “Training deep neural networks with 8-bit floating point numbers,” in Proceedings of the 32nd International Conference on Neural Information Processing Systems , 2018
2018
Earlier work this paper cites.
2018
Earlier work this paper cites.
R. Zellers, Y. Bisk, R. Schwartz, and Y. Choi, “SWAG: A large-scale adversarial dataset for grounded commonsense inference,” in Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , 2018
2018
Earlier work this paper cites.
——, “Nvidia-t4-datasheet,” 2019. [Online]. Available: https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Center/tesla-t4/t4-tensor-core-datasheet-951643.pdf
2019
Earlier work this paper cites.
D. Basu, D. Data, C. Karakus, and S. Diggavi, “Qsparse-local-sgd: Distributed sgd with quantization, sparsification and local computations,” Advances in Neural Information Processing Systems , vol. 32, 2019
2019
Cited alongside, same era.
A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala, “Pytorch: An imperative style, high-performance deep learning library,” in Advances in Neural Information Processing Systems , 2019
2019
Cited alongside, same era.
J. D. M.-W. C. Kenton and L. K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” in Proceedings of naacL-HLT , 2019
2019
Cited alongside, same era.
Z. Yao, Z. Dong, Z. Zheng, A. Gholami, J. Yu, E. Tan, L. Wang, Q. Huang, Y. Wang, M. Mahoney et al. , “Hawq-v3: Dyadic neural network quantization,” in International Conference on Machine Learning , 2021
2021
Later among the works it cites.
Y. Wu and J. Johnson, “Rethinking ”batch” in batchnorm,” ArXiv , vol. abs/2105.07576, 2021
2021
Later among the works it cites.
J. Chen, L. Zheng, Z. Yao, D. Wang, I. Stoica, M. Mahoney, and J. Gonzalez, “Actnn: Reducing training memory footprint via 2-bit activation compressed training,” in International Conference on Machine Learning , 2021
2021
Later among the works it cites.
X. Y. Geoffrey, Y. Gao, P. Golikov, and G. Pekhimenko, “Habitat: A { \{ Runtime-Based } \} computational performance predictor for deep neural network training,” in 2021 USENIX Annual Technical Conference (USENIX ATC 21) , 2021
2021
Later among the works it cites.
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
2019
Cited alongside, same era.
J. H. Park, G. Yun, C. M. Yi, N. T. Nguyen, S. Lee, J. Choi, S. H. Noh, and Y. ri Choi, “HetPipe: Enabling large DNN training on (whimpy) heterogeneous GPU clusters through integration of pipelined model parallelism and data parallelism,” in 2020 USENIX Annual Technical Conference (USENIX ATC 20) , 2020
2020
Cited alongside, same era.
L. Song, F. Chen, Y. Zhuo, X. Qian, H. Li, and Y. Chen, “Accpar: Tensor partitioning for heterogeneous deep learning accelerators,” in 2020 IEEE International Symposium on High Performance Computer Architecture (HPCA) , 2020
2020
Cited alongside, same era.
C. Chen, Q. Weng, W. Wang, B. Li, and B. Li, “Semi-dynamic load balancing: efficient distributed learning in non-dedicated environments,” Proceedings of the 11th ACM Symposium on Cloud Computing , 2020
2020
Cited alongside, same era.
Z. Bai, Z. Zhang, Y. Zhu, and X. Jin, “PipeSwitch: Fast pipelined context switching for deep learning applications,” in 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20) , 2020
2020
Cited alongside, same era.
X. Zhang, S. Liu, R. Zhang, C. Liu, D. Huang, S. Zhou, J. Guo, Q. Guo, Z. Du, T. Zhi, and Y. Chen, “Fixed-point back-propagation training,” in 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2020
2020
Cited alongside, same era.
F. Fu, Y. Hu, Y. He, J. Jiang, Y. Shao, C. Zhang, and B. Cui, “Don’t waste your bits! squeeze activations and gradients for deep neural networks via tinyscript,” in International Conference on Machine Learning , 2020
2020
Cited alongside, same era.
T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y. Jernite, J. Plu, C. Xu, T. L. Scao, S. Gugger, M. Drame, Q. Lhoest, and A. M. Rush, “Transformers: State-of-the-art natural language processing,” in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , 2020
2020
Cited alongside, same era.
A. M. Abdelmoniem and M. Canini, “Towards mitigating device heterogeneity in federated learning via adaptive model quantization,” in Proceedings of the 1st Workshop on Machine Learning and Systems , 2021
2021
Cited alongside, same era.
2022
Later among the works it cites.
V. Thakkar, P. Ramani, C. Cecka, A. Shivam, H. Lu, E. Yan, J. Kosaian, M. Hoemmen, H. Wu, A. Kerr, M. Nicely, D. Merrill, D. Blasig, F. Qiao, P. Majcher, P. Springer, M. Hohnerbach, J. Wang, and M. Gupta, “CUTLASS,” 2022
2022
Later among the works it cites.
X. He, J. Sun, H. Chen, and D. Li, “Campo: Cost-Aware performance optimization for Mixed-Precision neural network training,” in 2022 USENIX Annual Technical Conference (USENIX ATC 22) , 2022
2022
Later among the works it cites.
B. Xu, Y. Zhang, H. Lu, Y. Chen, T. Chen, M. Iovine, M.-C. Lee, and Z. Li, “AITemplate,” 2022. [Online]. Available: https://github.com/facebookincubator/AITemplate
2022
Later among the works it cites.
J. Xing, L. Wang, S. Zhang, J. Chen, A. Chen, and Y. Zhu, “Bolt: Bridging the gap between auto-tuners and hardware-native performance,” in Proceedings of Machine Learning and Systems , 2022, pp. 204–216
2022
Later among the works it cites.
H. Hu, C. Jiang, Y. Zhong, Y. Peng, C. Wu, Y. Zhu, H. Lin, and C. Guo, “dpro: A generic performance diagnosis and optimization toolkit for expediting distributed dnn training,” in Proceedings of Machine Learning and Systems , 2022
2022
Later among the works it cites.
Z. Liu, K. Zhou, F. Yang, L. Li, R. Chen, and X. Hu, “EXACT: Scalable graph neural networks training via extreme activation compression,” in International Conference on Learning Representations , 2022
2022
Later among the works it cites.