Fetching the paper…
Reading the bibliography…
We show communication schedulers' recent work proposed for ML collectives does not scale to the increasing problem sizes that arise from training larger models.
A Formal Basis for the Heuristic Determination of Minimum Cost Paths
Peter Hart, Nils Nilsson, and Bertram Raphael. 1968 · 1968
Earlier work this paper cites.
A survey of combinatorial optimization problems in multicast routing
Carlos AS Oliveira and Panos M Pardalos. 2005 · 1981
Earlier work this paper cites.
Multicast in the asynchronous transfer mode environment
John Matthew Simon Doar. 1993 · 1993
Earlier work this paper cites.
How bad is naive multicast routing?. In
M. Doar and I. Leslie. 1993 · 1993
Earlier work this paper cites.
The communication challenge for MPP: Intel Paragon and Meiko CS-2
Roger W Hockney. 1994 · 1994
Earlier work this paper cites.
Optimum routing of multicast streams. In
C.A. Noronha and F.A. Tobagi. 1994 · 1994
Earlier work this paper cites.
Introduction to linear optimization
Dimitris Bertsimas and John N Tsitsiklis. 1997 · 1997
Earlier work this paper cites.
Convex Optimization
Stephen Boyd and Lieven Vandenberghe. 2004 · 2004
Earlier work this paper cites.
The Floyd–Warshall algorithm on graphs with negative cycles
Stefan Hougardy. 2010 · 2010
Earlier work this paper cites.
Inter-Datacenter Bulk Transfers with Netstitcher. In
Nikolaos Laoutaris, Michael Sirivianos, Xiaoyuan Yang, and Pablo Rodriguez. 2011 · 2011
Cited alongside, same era.
Optimization with gurobi and python
Joo Pedro Pedroso. 2011 · 2011
Cited alongside, same era.
Achieving High Utilization with Software-Driven WAN. In
Chi-Yao Hong, Srikanth Kandula, Ratul Mahajan, Ming Zhang, Vijay Gill, Mohan Nanduri, and Roger Wattenhofer. 2013 · 2013
Cited alongside, same era.
B4: Experience with a globally-deployed software defined WAN
Sushant Jain, Alok Kumar, Subhasree Mandal, Joon Ong, Leon Poutievski, Arjun Singh, Subbaiah Venkata, Jim Wanderer, Junlan Zhou, Min Zhu, et al · 2013
Cited alongside, same era.
Calendaring for Wide Area Networks. In
Srikanth Kandula, Ishai Menache, Roy Schwartz, and Spandana Raj Babbula. 2014 · 2014
Cited alongside, same era.
Bert: Pre-training of deep bidirectional transformers for language understanding
Synthesizing Optimal Collective Algorithms. In
Zixian Cai, Zhengyang Liu, Saeed Maleki, Madanlal Musuvathi, Todd Mytkowicz, Jacob Nelson, and Olli Saarikivi. 2021 · 2021
Later among the works it cites.
DeepLight: Deep Lightweight Feature Interactions for Accelerating CTR Predictions in Ad Serving. In
Wei Deng, Junwei Pan, Tian Zhou, Deguang Kong, Aaron Flores, and Guang Lin. 2021 · 2021
Later among the works it cites.
Solving Large-Scale Granular Resource Allocation Problems Efficiently with POP. In
Deepak Narayanan, Fiodar Kazhamiaka, Firas Abuzaid, Peter Kraft, Akshay Agrawal, Srikanth Kandula, Stephen Boyd, and Matei Zaharia. 2021 · 2021
Later among the works it cites.
TACCL: Guiding Collective Algorithm Synthesis using Communication Sketches
Aashaka Shah, Vijay Chidambaram, Meghan Cowan, Saeed Maleki, Madan Musuvathi, Todd Mytkowicz, Jacob Nelson, Olli Saarikivi, and Rachee Singh. 2021 · 2021
Later among the works it cites.
Campion: Debugging Router Configuration Differences. In
Alan Tang, Siva Kesava Reddy Kakarla, Ryan Beckett, Ennan Zhai, Matt Brown, Todd Millstein, Yuval Tamir, and George Varghese. 2021 · 2021
Later among the works it cites.
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018 · 2018
Cited alongside, same era.
Blink: Fast and generic collectives for distributed ml
Guanhua Wang, Shivaram Venkataraman, Amar Phanishayee, Nikhil Devanur, Jorgen Thelin, and Ion Stoica. 2020 · 2020
Cited alongside, same era.
Contracting wide-area network topologies to solve flow problems quickly. In
Firas Abuzaid, Srikanth Kandula, Behnaz Arzani, Ishai Menache, Matei Zaharia, and Peter Bailis. 2021 · 2021
Cited alongside, same era.
Better Together: Jointly Optimizing ML Collective Scheduling and Execution Planning using SYNDICATE
Kshiteej Mahajan, Ching-Hsiang Chu, Srinivas Sridharan, and Aditya Akella. [n. d.]
Cited in the paper.
([n. d.])
MSCCL codebase [n. d.]
Cited in the paper.
Themis: A Network Bandwidth-Aware Collective Scheduling Policy for Distributed Training of DL Models. In
Saeed Rashidi, William Won, Sudarshan Srinivasan, Srinivas Sridharan, and Tushar Krishna. 2022 · 2022
Later among the works it cites.
TopoOpt: Co-optimizing Network Topology and Parallelization Strategy for Distributed Training Jobs
Weiyang Wang, Moein Khazraee, Zhizhen Zhong, Manya Ghobadi, Zhihao Jia, Dheevatsa Mudigere, Ying Zhang, and Anthony Kewitsch. 2022 · 2022
Later among the works it cites.
Optimal Direct-Connect Topologies for Collective Communications
Liangyu Zhao, Siddharth Pal, Tapan Chugh, Weiyang Wang, Prithwish Basu, Joud Khoury, and Arvind Krishnamurthy. 2022 · 2022
Later among the works it cites.