A key challenge in autoregressive decoding is that performance is often limited more by memory bandwidth and data movement than by raw compute, making GPUs, which are optimized for massive parallelism, a suboptimal fit. The Grog 3 LPU is designed specifically to address this mismatch.

In its most advanced form, NVIDIA implements Attention-FFN Disaggregation (AFD), splitting even the decode phase: GPUs handle attention over the KV cache while LPUs execute stateless FFN/MoE layers at high bandwidth with deterministic latency. NVIDIA's Dynamo platform orchestrates a per-token handoff across both engines.

This matters most for agentic AI workloads – long contexts, continuous multi-turn generation, and strict latency requirements. NVIDIA claims that an LPX rack paired with Vera Rubin NVL72 can deliver up to 35x higher throughput per megawatt for trillion-parameter-scale models, compared to a standalone Blackwell NVL72 system.

What is the Grog 3 LPU?

The Grog 3 LPU is built on Groq's Tensor Streaming Processor (TSP) – a dataflow architecture where data streams through compute units rather than shuttling between memory and registers as in conventional Von Neumann GPUs. This eliminates load/store bottlenecks, yielding higher utilization for bandwidth-bound workloads.

The LP30 chip carries 512 MB of on-chip SRAM per die with 150 TB/s memory bandwidth – nearly 7x the 22 TB/s from Rubin GPU's 288 GB HBM4. The SRAM capacity is small by comparison, but the bandwidth advantage is decisive for memory-bound decoding. A full LPX rack (256 LP30 chips) delivers 128 GB SRAM, 40 PB/s aggregate bandwidth, and 315 PFLOPS FP8.

Key Features

• SRAM-only memory: 512 MB on-chip SRAM per chip provides significant higher bandwidth than HBM, releasing the memory wall for decoding – but limited capacity means trillion-parameter models require multiple racks with weights distributed across the 256-chip fabric.
• Dataflow execution: Compiler-orchestrated, deterministic data movement replaces GPU-style dynamic thread scheduling, delivering predictable per-token latency critical for strict SLAs.
• KV cache handling: The LPU uses a flat, cacheless SRAM architecture – no hierarchy, no prefetch, all data placement resolved by the compiler at compile time. In AFD mode, the LPU executes only stateless FFN/MoE work; KV cache storage and attention computation remain on the GPU side in HBM, with small activation tensors exchanged per token via All-to-All collectives.

In summary, the LPU is best understood as a dedicated, low-latency decode accelerator within a heterogeneous inference pipeline for large-scale AI systems – a specialized co-processor that works alongside GPUs to optimize end-to-end inference performance.

What Makes it Unique?

How the LPU Replaces Von Neumann's Fetch-Execute Cycle

The TSP architecture fundamentally differs from Von Neumann processors (CPUs and GPUs), where a fetch-decode-execute cycle introduces unpredictable latency through branch prediction, out-of-order execution, cache misses, and thread contention. GPUs add further variability via dynamic resource partitioning and runtime warp scheduling.

The TSP strips all of this away – no branch predictors, arbiters, reorder buffers, or traditional caches. Instead, it uses a functionally sliced design: memory and compute units are physically interleaved into a programmable assembly line. Data streams through in a fixed pattern, with each slice performing its operation as data passes through. Rather than pulling data to the compute (Von Neumann), the compute is organized around the data flow – yielding deterministic, predictable performance.

One Giant Core: Functionally Sliced Microarchitecture

Unlike GPUs with thousands of small cores, the TSP is a single large core with alternating bands of memory (SRAM) and compute (vector/matrix units). Data streams from one band to the next with minimal distance, exploiting the fact that in AI workloads, each operation's output feeds directly into the next.

The main components: SRAM slices store model weights directly (not as cache), under full compiler control with guaranteed access latency – no variable cache hit/miss behavior. The 512 MB across many banks enables 150 TB/s bandwidth. Matrix units (MXM) handle core tensor math like matrix-vector multiplications. These are intentionally small relative to SRAM, since decoding is bottlenecked by data delivery, not compute. Vector units (VXM) perform activations, normalization, and element-wise operations between matrix layers. Stream registers connect slices together as high-bandwidth conduits, with the compiler ensuring data arrives at each stage at exactly the right clock cycle.

Role in the Vera Rubin Rack Architecture

The LP30 does not operate independently – it is one of seven chips in the Vera Rubin architecture (alongside Vera CPU, Rubin GPU, NVLink 6, ConnectX-9, BlueField-4, and Spectrum-6), each handling a specific inference pipeline stage. This reflects NVIDIA's shift from optimizing individual chips to optimizing the entire system.

The Vera Rubin rack assigns distinct roles to its constituent components:

Three-stage Inference Pipeline

• Prefill (GPU): The input prompt is processed in parallel on the GPU, which excels at this embarrassingly parallel phase. Output: a full KV cache of the input's contextual state.
• Decode (AFD Loop): Each token cycles between two engines – the GPU computes attention over the KV cache, hands the resulting activation to the LPU via high-speed interconnect, the LPU executes FFN/MoE layers at extreme bandwidth, then returns the result for the next layer's attention. This repeats per layer, per token.
• Orchestration (Dynamo): NVIDIA's Dynamo platform manages the full pipeline – request routing, batch/latency-aware scheduling, KV cache lifecycle, and per-token handoff between GPU and LPU.

Solving the KV Cache Explosion

AFD helps mitigate this bottleneck by decoupling attention and FFN execution across different processors. In NVIDIA’s implementation, KV cache-intensive attention remains on the GPU, while the LPU handles stateless FFN/MoE computation.

Because attention stays on GPUs with expandable HBM, the LPU is unaffected by context length. Its SRAM stores only fixed FFN or expert weights, enabling the same LPX rack to support contexts from short to very long contexts without any hardware changes on the LPU side.

LPX Host CPU: Evolution Path

A key question is whether the LPX host CPU will shift from x86 to NVIDIA’s Arm-based Vera (or its successor, Rosa), completing Jensen Huang’s vision of a fully vertically integrated platform. In the near term, this transition is unlikely. The current LPX architecture depends on an x86 CPU’s mature PCIe root complex, while Vera is optimized for NVLink C2C. Moving to Arm would require major redesigns (e.g. FPGA bridging or LPU interconnect changes) and software porting, with limited benefit for a control-plane CPU.

Through 2027, changes are expected to be incremental, with the existing architecture largely intact. The real inflection point is likely around 2028 (LP40 generation), when NVLink C2C could be integrated directly into the LPU. At that stage, NVIDIA could eliminate the FPGA bridge and x86 CPU, replacing them with a fully in-house stack (Rosa CPU + LPU + NVLink + BlueField), creating a clean, end-to-end NVIDIA platform.

Strategically, this shift matters. As long as LPX depends on x86, it remains an add-on module within the Vera Rubin system. A full transition would signal that the Groq acquisition has been fully absorbed, enabling tighter software integration, a simplified supply chain, and a differentiated, fully integrated inference platform.

Market Positioning: Not Competing with Low-end ASICs (Yet)

A critical aspect of NVIDIA's LPU strategy is its market positioning in the premium token tier ($45-$150 per million tokens), where ultra-low latency and high interactivity justify the cost of dedicated SRAM hardware. Competitors like Cerebras, which pairs its 44 GB wafer-scale SRAM chip with AWS Trainium in a similar disaggregated architecture. Post-acquisition, the Groq 3 LPU operates as an accelerator within NVIDIA's CUDA stack on the Vera Rubin platform rather than as a standalone inference product; it does not natively support CUDA, with orchestration handled entirely by Dynamo. However, the GroqCloud entity remains operational and has secured a $1.5 billion contract to build LPU-based data centers in Saudi Arabia, indicating that Groq-branded inference services continue alongside NVIDIA's integrated platform offering.

Initial Target Segment

The first generation of LPUs is targeted squarely at the premium inference market:

• Trillion-parameter models: Workloads involving MoE models at the trillion-parameter scale (e.g. DeepSeek V3 class), where the memory bandwidth requirements of decode are most acute and the AFD architecture delivers the greatest throughput-per-watt advantage. NVIDIA claims 35x higher throughput per megawatt versus Blackwell NVL72 alone.
• Performance-first deployments: Use cases where latency, throughput, and output quality outweigh cost-per-token – such as enterprise AI assistants, real-time code generation, and autonomous agent systems. While market discussions have suggested potential pricing benchmarks, NVIDIA has not disclosed any official token-level pricing.
• Integrated NVIDIA ecosystems: Customers have already invested in NVIDIA’s GPU and networking infrastructure, where the LPX rack can be deployed as an optional addition to Vera Rubin NVL72 configurations without requiring wholesale infrastructure changes.

Long-term Expansion Strategy

NVIDIA’s roadmap indicates that LPU will follow a classic top-down market penetration strategy. As manufacturing costs decline through process maturation and volume scaling, and as the Dynamo software ecosystem around heterogeneous inference matures, the LPU product line is expected to expand into mid-range inference markets. One notable advantage of the LPU strategy is supply chain independence: by relying on Samsung foundries and on-chip SRAM rather than TSMC capacity and HBM, the LPX racks represent truly incremental production capacity that does not cannibalize NVIDIA’s existing GPU allocation.

Final Thought

The next phase of AI competition will not be defined solely by larger models, but by who builds the infrastructure for continuous, autonomous intelligence. As AI shifts from prompt-driven interaction to persistent, agentic execution, the bottleneck moves from compute to system design – from models to infrastructure. In that transition, architectures like NVIDIA’s LPU signal a fundamental shift: AI scaling is no longer just about training capability, but about sustaining real-time, long-context, always-on inference.