As data and model complexity grow, centralized training encounters GPU storage and computing constraints. To address these challenges, Distributed Training (DT) has become a common approach as it leverages parallelization (e.g., of data and tensors) as well as dedicated network communication libraries, referred to as Collective Communications (CCs). However, this shift introduces new networking challenges, such as how to efficiently exchange packets between GPUs to reduce in-network congestion and how to tailor CCs to the underlying data center fabric. This thesis will first shed light on the main parallelization techniques and how they address the limitations of centralized approaches. Next, we will delve into Collective Communication Libraries (CCL), providing an overview of the most advanced existing solutions, such as NCCL and TECCL.