Implications of Scaling Up

Scaling up means increasing the compute density of an AI system, essentially by increasing the number of interconnected GPUs in each AI server and in a whole rack. In a rackscale system, GPUs are interconnected by means of a compute fabric that transfers data at hundreds of GB/s and nanosecond-scale latency, with communication and synchronization happening across all GPUs in the rack. This is typically facilitated by a copper-based scale-up network, which effectively ensures that all GPUs in the rack function as one massive coherent GPU.

Rackscale systems represent a paradigm shift in AI compute and have had a transformative impact on the AI market by establishing new standards for performance, scale and efficiency in AI computing. However, they are extremely complex computing systems, with major hardware and quality control challenges, particularly with respect to power and cooling requirements. Compared to conventional “node-based” computing, the transition to coherent compute rackscale systems involves unheard of levels of precision with respect to the manufacturing of high-speed interconnects, integration of components as well as unprecedented levels of quality control testing of fully assembled racks.

Variations in Interconnect Properties

At very high data rates, even the minutest imperfections in component manufacturing and the quality of materials used for high-speed interconnects can become significant problems. For example, small variations in a copper trace’s width or thickness in a PCB, inconsistencies in a solder joint or microscopic differences in a PCB’s dielectric composition can affect signal integrity, as can impedance mismatches due to tiny differences in the electrical resistance and capacitance along a signal path. Even minute differences in the final length or routing of individual copper wires in a bundle can cause jitter and skew increasing latency.

This means that the physical and electrical uniformity of every interconnect in a scale-up network is critical as any variance due to manufacturing imperfections, impedance mismatches, etc. can impact a signal’s integrity or increase latency, which in turn, can severely impact AI model performance and significantly drive up rack power consumption. As a result, modern rackscale systems are developed as integrated systems, where the physical components (PCBs, connectors, cables, etc.) need to be engineered with micron-level precision to ensure that the rack functions as a single, unified compute system with predictable, high-quality signal transmission. Counterpoint Research believes that this is one of the most underappreciated challenges of AI hardware engineering at present and a problem faced by all AI server component vendors and hyperscalers as the industry transitions from “node-based” AI compute to large-scale coherent compute based rackscale systems.

Figure 1 shows the various connectivity chips found in a typical AI server together with the main silicon vendors.

Figure 1: Schematic overview of AI server interconnect chips and vendors (pink boxes)

Quality Control Challenges

Traditional cloud computing racks typically operate at power densities between 5kW and 15kW, but AI workloads, driven by the use of GPUs, push these requirements much higher—up to 120kW per rack in the case of NVIDIA’s GB200 NVL72 system and 140kW for the latest GB300 NVL72 system. Upcoming rackscale designs such as NVIDIA’s Kyber rack and AMD’s Helios rack will push rack density to 700 kW and beyond making the design and quality control testing of scale-up networks even more challenging.

In addition to testing individual servers, rackscale quality control involves systems level integration tests encompassing rack-level validation as well as continuous operational quality control throughout the lifetime of the rack. Future increases in rack power density will necessitate the development of new and more stringent quality control processes and standards, including the extensive use of micron-precision non-destructive testing procedures. As a result, Counterpoint Research believes that the use of advanced, high-precision metrology equipment will become mandatory and mainstream for all AI server and rack vendors over the next few years.

[1] Counterpoint Report “Scaling-Out AI Factories – Key Vendors and Technologies” discusses the importance of high-speed scale-out networking in AI Data Centers.

Counterpoint Research’s report “Scaling Up AI Racks – Key Vendors and Technologies” analyses the key connectivity technologies of a scale-up network and includes key chip and component vendors – both major established players and emerging start-ups - offering connectivity for scale up networks.

Report Table of Contents

• Introduction
• Chiplet Die-to-Die
• Compute Fabric
• GPU-to-GPU Connectivity
• GPU-CPU Connectivity
• CXL Memory
• PCIe Switches
• PCIe Cards
• PCIe Retimers
• DAC and AEC Cables
• Passive DAC vs AEC
• AEC Retimers
• AEC DSPs
• Analyst Viewpoint