Revolutionizing AI Training: The Evolution of RoCE Networks for Distributed Scalability
Image Credit: Alina Grubnyak | Unsplash
The surge in AI applications has dramatically increased the demands on data center networks, particularly for distributed AI training which involves tight synchronization across thousands of GPUs. This requirement for high-performance networking is crucial as AI jobs, like training generative models, can span several weeks and involve intricate data flows. The need for efficient, scalable solutions has led companies like Meta to reevaluate and redesign their network infrastructure specifically for these high-load scenarios.
The Rise of RoCE in AI Networks
To meet the demands of distributed GPU training, Meta adopted RDMA Over Converged Ethernet version 2 (RoCEv2) for its data center networks. RoCEv2 offers high bandwidth, low latency, and lossless data transfer, making it ideal for the intense communication needs of AI training clusters. This technology supports the seamless coordination required among GPUs, facilitating efficient data processing and model training across extensive network architectures.
Designing Dedicated AI Training Networks
Meta constructed specialized backend networks to support distributed training, allowing these segments to operate independently from the main data center network. This strategic separation ensures that AI training jobs do not interfere with other data center operations, optimizing both training efficiency and general network performance. The backend networks are specifically tailored to handle large-scale AI workloads, incorporating deep understanding of job requirements and network demands.
Network Topology and Infrastructure
The training network topology at Meta is designed with a dual-network structure: a frontend (FE) network handling tasks like data ingestion and logging, and a backend (BE) network focused on the training process itself. This structure allows for efficient data management and supports the high-throughput requirements of AI training. The backend is equipped with RoCEv2 technology, ensuring that inter-GPU communication is fast and reliable, critical for maintaining the pace and accuracy of model training.
Advanced RoCE Configurations for Scalability
Meta has advanced the configuration of its RoCE networks by implementing a fabric-based architecture. This includes a two-stage Clos topology, ensuring scalability and redundancy within the AI training clusters. The architecture features rack training switches (RTSW) connected to cluster training switches (CTSW) via high-speed links, supporting thousands of GPUs with high-capacity, non-blocking connectivity.
Overcoming Topological Limitations
The initial deployment of simple star topologies revealed limitations in scale and redundancy, prompting a shift to more complex, scalable network designs. By integrating a spine-leaf architecture in the AI zones, Meta has enhanced its network’s ability to grow and support increasingly larger AI models and training requirements. These zones are interconnected through a high-capacity backbone, enabling efficient communication across distributed systems.
Routing and Traffic Management Strategies
Efficient routing and traffic management are critical in large-scale AI training networks. Meta initially employed ECMP but shifted to a path-pinning strategy to better manage the unique traffic patterns of AI workloads, which include low entropy and high burstiness. Adjustments to these strategies were necessary to address issues of congestion and uneven traffic distribution, which could significantly impact training performance.
Implementing Congestion Control Mechanisms
To handle the intense and bursty traffic flows characteristic of AI training, Meta has experimented with various congestion control mechanisms. The transition to 400G deployments necessitated further adjustments in congestion management strategies to maintain network performance. The use of PFC (Priority Flow Control) and adjustments in ECMP settings are examples of how Meta has tailored its approach to meet the specific demands of AI training traffic.
Innovations in Receiver-Driven Traffic Admission
Meta has developed a receiver-driven traffic admission system to further refine how data flows through its AI training networks. This system enhances the network's ability to manage congestion by controlling the rate at which data is sent based on receiver capacity. Such innovations are crucial for maintaining system stability and performance, especially as network speeds and scales increase.
