The rapid expansion in the size and complexity of AI workloads is significantly impacting both compute and networking technologies used in modern data centers. Al clusters are limited to around 100k-200k GPUs today but are expected to increase to 1 million GPUs and beyond during the next two to three years. This will have a significant impact on the design and bandwidth requirements of the back-end network used to provide GPU-to-GPU communications across hundreds or thousands of racks.

Smart Networking is Critical

Companies are investing at an unprecedented scale in the latest cutting edge GPUs to accelerate training and inference. However, expensive AI clusters can underperform not because of insufficient compute but due to an inefficient and poorly performing back-end networks. Distributed training and inference workloads require frequent high-throughput, low-latency data exchanges between GPUs. If these data exchanges are slow due to data congestion, GPUs will sit idle and will be underutilized, resulting in longer training cycles or inconsistent inference performance.

AI at scale is therefore not a compute problem, but a systems level engineering challenge with networking at the heart of it. Small inefficiencies will be amplified across a large cluster, which in turn delays time-to-market, escalates power and operational costs and makes predictable bandwidth and congestion control indispensable. By eliminating data bottlenecks, boosting utilisation and delivering predictable performance, smarter networking enables more predictive AI with a faster time to insights and a better ROI on GPU infrastructure.

Converged Compute and Networking Fabrics

The scale-out networking market today is split into two camps: Nvidia with its proprietary Infiniband and a small number of fast, vendor-specific Ethernet derivatives, for example, Nvidia’s SpectrumX - forerunners of the Ultra Ethernet Consortium (UEC)’s upcoming super-fast open standards-based Ethernet protocol. However, this is a market that is rapidly changing with several vendors proposing fabrics that combine scale-up and scale-out into a single networking standard. In fact, this is a major trend in AI data center and HPC networking right now.

Broadcom, for example, recently announced its Scale-Up Ethernet (SUE) fabric, which the company claims can be used both as a scale-up and scale-out fabric, depending upon the application. There is also the Ethernet Scale-Up Network (ESUN) protocol, a new initiative by the Open Compute Project (OCP) to push Ethernet to meet the more demanding connectivity, latency and high-bandwidth requirements of scale-up infrastructure. ESUN complements - and in some respects - overlaps with SUE and UEC. However, both compete against Nvidia’s NVLink and the upcoming UALink standard.

Yet another option comes from start-up Astera Labs based on the PCIe Express standard. Although traditionally used for scale-up networking, Astera describes its Scorpio-P chip as the industry’s first PCIe fabric switch purpose built for scale-out and complements its Scorpio-X switch designed for scale-up. Astera therefore offers scale-up and scale-out via a single fabric. This will enable AI data center operators to use PCIe to connect GPUs within and across a few racks, i.e. within a pod - although presumably Ethernet will still be used for pod-to-pod and cross data center networking.

Dynamic Vendor Ecosystem

The vendor market for scale-out connectivity silicon is also fast moving, very competitive and increasingly strategic. Large scale deployments today are dominated by a handful of incumbents with major hyperscale partners, for example, Broadcom. However, the convergence of compute and networking fabrics means that this is a very dynamic market with lots of opportunities for innovation. As a result, there are a number of niche start-ups as well as some legacy platform providers challenging the incumbents (Figure 1). This will accelerate with the deployment of optical interconnects within the next two years or so. Although the barriers to wide adoption are high, innovative technology can scale quickly if the technology finds favour with key customers. Inevitably, the hyperscalers will play a critical role in deciding who wins and who loses!

Figure 1: Scale-Out Networking Showing Key Components And Vendors

Counterpoint Research report “Scaling-Out AI Factories – Key Vendors and Technologies” provides an overview of the main components of an AI data centre’s scale-out network and highlights the key players - both big tech and start-ups - that provide the crucial connectivity chips and hardware to enable high-performance rackscale based AI computing across hundreds or thousands of racks.

Table of Contents

Introduction
Clos Leaf and Spine Network Architecture
Leaf Switch
Spine Switch
Network Interface Card (NIC)
Pluggable Optical Modules
Form Factors
NRZ vs PAM4 Modulation
Optical Module Supply Chain
Optical Module Integrators
Laser & Optoelectronic Vendors
DSP ASIC Vendors
Data Center Interconnect (DCI)
Short-Distance DCI
Long-Distance DCI

Author