Flow Scheduling for Collective Communications in Data Centers

As data and model complexity grow, centralized training encounters GPU storage and computing constraints. To address these challenges, Distributed Training (DT) has become a common approach as it leverages parallelization (e.g., of data and tensors) as well as dedicated network communication libraries, referred to as Collective Communications (CCs). However, this shift introduces new networking challenges, such as how to efficiently exchange packets between GPUs to reduce in-network congestion and how to tailor CCs to the underlying data center fabric. This thesis will first shed light on the main parallelization techniques and how they address the limitations of centralized approaches. Next, we will delve into Collective Communication Libraries (CCL), providing an overview of the most advanced existing solutions, such as NCCL and TECCL. As the complexity of these DT systems is hard to model, it is preferable to rely on a dedicated testing environment, whose implementation details will be discussed in this thesis. Finally, we will provide an overview on the opening research directions as well as the ongoing work that I'll continue addressing during my PhD.