Advancements in Distributed Systems for Large Language Model Training and Serving

The rapid advancements in large language models (LLMs) have revolutionized the field of artificial intelligence, enabling break- throughs in natural language processing, generation, and reasoning. However, the exponential growth in model size and computational requirements poses significant challenges in efficiently training and serving these models. This paper presents a comprehensive review of recent advancements in distributed systems for training and serving LLMs, highlighting key techniques, frameworks, and systems that address the scalability, efficiency, and fault tolerance challenges. For distributed training, we discuss various paralleliza- tion strategies, including data, model, and pipeline parallelism, and their integration into systems like Megatron-LM and DeepSpeed. We focus on novel approaches such as ZeRO, 3D parallelism, and SWARM parallelism, which enable training of models with billions to trillions of parameters. Techniques for optimizing communica- tion, load balancing, and fault tolerance, such as asynchronous training, and efficient checkpointing, are also explored. In the do- main of serving, we examine systems and methods that support efficient inference, including model quantization, distillation, and optimization frameworks such as TensorRT and ONNX Runtime. Additionally, we review case studies and real-world applications, providing insights into the deployment and operational challenges faced by industry leaders. Our survey aims to provide a holistic understanding of the state-of-the-art in distributed training and serving of LLMs, identifying key research directions and open chal- lenges for future exploration.