Report

Memory Solutions for Gen AI Part 31: KV Cache Meets Memory Wall

May 4, 2026

Artificial intelligence algorithms, such as Gemini and ChatGPT, need a large memory storage called the key-value (KV) cache for long conversations. This is especially true with the rise of agentic AI. Memory usage is growing rapidly, creating a new "memory wall."

Software companies are looking for ways to reduce memory use. However, supply does not meet demand. Buyers are signing multi-year contracts at prices that are many times higher than last year. Downstream industries, such as smartphone makers, must actively seek survival. Investors should be less concerned about arguments that the market has already peaked despite the record high profit margin.

Key Takeaways

Increasing level of KV cache during inference will continue to drive polynomial growth in memory demand.
Demand prediction remains challenging with hierarchical data tiering from HBM to DRAM and SSD, depending on specific LLMs.
Software optimization is a risk in memory demand but also act as a catalyst for expanding the areas of memory utilization.
Trade ratio between supply of commodity memory and AI memory gets worse, extending the pricing upside from HBM to DRAM/ NAND.

KV Cache continue to drive polynomial growth in memory demand

KV Cache in Inference

The process of generating a response in an LLM (Large Language Model) is called “Inference”. The model builds text by predicting the next token one by one based on past tokens. To generate the 100th token, the model needs to re-evaluate the relationship with the preceding 99 tokens. In other words, if you generate an output with n tokens, each new token must be compared with the previous n-1 tokens, causing the computational load to increase in proportion to n^2. As the response grows longer, computational power and memory usage increase polynomial.

However, what if we retrieve the KV caches that have already been calculated? When generating an n token output, the model only performs operations for the final n^th token and reuses the other values. Consequently, even as the number of tokens n grows, the computation increases linearly.

The origin of KV Cache – Transformer. An LLM is an AI model designed to understand and generate language like a human by learning from vast amounts of data. Most current LLM models, such as ChatGPT and Gemini, are based on the Transformer model introduced by Google in 2017.

Mathematical approach and examples. The Transformer model calculates the relationships between all words in a sentence, allowing it to pay more "Attention" to other contextually important tokens when processing a specific token. The Attention value represents the correlation between the vectors of Q, K, and V. By calculating their dot-product and applying value weights, the model understands the context of a token and completes the sentence. This relationship can be expressed through the following formula:

Attention(Q,K,V)=softmax((QK^T)/√(d_k ))V

Q (Query): A question about a specific word—"What am I looking for?"
K (Key): What other words provide—"What do I offer?"
V (Value): The content itself—"What is my actual information?"
Scaling (√(d_k ) ) & softmax: The elements for balancing the scores and picking the most important words to focus on.

This formula calculates the correlation between input data to quantify the weight of attention required for other words within a sentence when understanding a specific word. Consequently, it assigns high weights to information of high importance and dilutes information of lower importance, enabling a multi-dimensional understanding of the context.

Let’s find out what "it" refers to in the sentence:

"An apple fell from the tree. It is red." Here, the subject posing the question, Q, becomes "it." The model then evaluates the relationship between "it" and other words in the sentence (Apple, tree, red) by substituting them into the K (Key) matrix.
Through dot-product operations with the values in K, the model calculates a score to see which word yields the highest relevance. Since "apple" is the best match for "it," it is assigned the highest value, while other words receive lower scores. Finally, by applying weights to these results, "it" becomes connected to the attributes possessed by "apple." Now, let’s consider the relationship between K and V. In this sentence, "it" refers to "apple," and the fact that an apple is a red fruit will not change even if subsequent questions follow. These pre-computed K-V values are reused, and this stored data is known as the KV Cache.

How Large is the KV Cache?

The size of the KV cache required for a single token can be calculated based on the model's architecture like following.

KV Size per token=2 ×Layer ×Dimension ×Precision
2: The number of vectors (K, V)
Layer: The total number of layers (depth) of the model
Dimension: The size of the hidden vector for each token
Precision: The number of bytes used for the data type

Let’s take Google’s Gemini 3.1 Pro as an example:

To provide premium service, we can assume it uses a high number of layers and dimensions. If we assume the model has 80 layers, a dimension of 8192, and uses BF16 (2 bytes) for data, the KV cache per token would be approximately 2.5 MB. However, Gemini 3.1 Pro is estimated to use about 0.32 MB per token because it employs Grouped-Query Attention (GQA), a technique where multiple query heads (e.g. 8 heads) share a single KV cache.
What happens if we provide an entire book and ask for a summary? Or if we provide a 30-minute video and ask the model to compress it into roughly 1,000 tokens? In the case of Gemini 3.1 Pro, it can process up to 1 million tokens as input. If this context window is fully utilized, the input phase alone requires approximately 320 GB of KV cache (0.32MB x 1M). If 100 users were in the same situation simultaneously, it would require 32 TB of cache (320GB x 100). As the lengths of input and output increase, the KV cache grows rapidly, which in turn increases the amount of required memory. Therefore, as memory demands grow, efficient usage becomes essential.
Gemini 3.1 Pro allows up to 1 million tokens for input but limits the output to around 65,536 tokens to prevent the cache from becoming excessively large. While the number of tokens directly impacts memory volume, this limitation also stems from the structure of the Transformer model: input can be processed in parallel for KV operations, whereas output must be calculated sequentially for every word generated.

Is the demand for KV cache predictable? On the server side, predictions are made based on CPU/GPU workloads. However, it varies depending on which specific LLM model is deployed.

Fundamental uncertainties are added to demand forecasting because one must choose between

Optimizing the LLM to fit the given hardware
Purchasing hardware to match the desired performance of a specific LLM. Furthermore, deciding whether to provision for the worst-case scenario (maximum input - maximum output) or the usual case (average input - average output) makes demand prediction even more challenging.

Above all, memory usage for the key-value (KV) cache fluctuates wildly based on concurrent users and computational complexity. This total volume of the KV cache residing in memory grows exponentially in advanced AI applications. One driver is chain-of-thought processing, where systems iterate to find correct answers. Another driver is multi-agent systems, where several artificial intelligence entities converse and collaborate.

Hierarchical structure of memory: Where to store the KV Cache

Memory within a processor system is structured in a hierarchy from SRAM to external storage devices, managing data in small units called pages. When an LLM operates, the program loads the KV Cache into the memory with the fastest accessibility. Data that is used less frequently is gradually moved to memory tiers with lower accessibility.

Ideally, the KV Cache is stored in HBM located directly next to the GPU.
However, once the HBM is full, less active KV Caches are sent to the DRAM or SSD and retrieved later when needed.
Larger HBM and DRAM capacities allow the system to hold more pages and deliver them quickly to the processor, resulting in faster response times.
Conversely, when capacity is reached, the system slows down as it frequently moves less-used pages out (page-out), swaps in stored content (page-in), or repeat re-computations.

The types of memory used for KV Cache storage are as follows:

SRAM (Static RAM): It maintains data as long as power is supplied, offering high power efficiency and speed. However, it is expensive and has relatively low density compared to its physical size. While primarily used as caches near the processor cores, inference-specific chips featuring large amounts of built-in SRAM have recently emerged to minimize latency to the extreme.

HBM: Attached directly to the GPU/ASIC, it can achieve speeds of several TB/s, making it ideal for handling the KV Cache. However, it is expensive, limited in capacity, and cannot be expanded after manufacturing.

DRAM (Dynamic RAM): The most common type of server memory, requiring periodic power refreshes. While DRAM can handle the KV Cache, its speed is significantly lower than HBM. Therefore, it serves as a buffer between HBM and SSD. DDR is currently the most widely used in servers; when used with CXL (Compute Express Link), it allows for easy memory expansion and sharing across multiple processors. LPDDR is emerging as a server memory due to its low power consumption. Although typically fixed to boards via BGA (Ball Grid Array)—making replacement difficult—it is now being used in detachable formats like SOCAMM or LPCAMM. When packaged with a CXL controller chip, it can also be expanded via CXL.

SSD (Solid State Drive): Although rarely used for active processing due to slower I/O speeds compared to DRAM, its low cost and non-volatile nature make it a popular storage space for offloading. It stores "past conversations" or inactive KV Caches that are not immediately needed. Recently, using the Boot Drive (intended for OS loading) for KV Cache has become more common; despite its smaller size (around 1TB), it guarantees high I/O priority and low latency within the system.

Exploration of KV Cache Storage Efficiency through SW optimization

As the processing of multimodal data—including video and images—increases alongside text, the management of large-scale multimodal caches will require more granular layering and compression tailored to specific data characteristics. The quest for software improvements, such as "How can we further reduce the KV Cache?", "How can we manage it more efficiently?", and "What are the optimal parameters?", represents the essence of inference efficiency—aiming to provide better services while utilizing limited resources.

While some argue that such software development could lower the demand for hardware like chipsets or memory, in the long run, it will likely expand the demand for a diverse range of hardware. Efficient software accelerates On-Device AI, and the evolution of software that utilizes hardware more effectively becomes a powerful driver for high-specification hardware to enable even more sophisticated AI experiences. Ultimately, software optimization will act as a catalyst for expanding the areas of memory utilization rather than merely reducing memory demand.

The core ideas behind these optimizations involve focusing on what is important and maintaining high speeds through efficient memory management.

Efficient Management: As pages are repeatedly evicted and filled, fragmentation occurs, leaving empty slots scattered throughout the memory. By breaking the KV Cache into smaller units—matching the memory's native page size—the system can fill any available page.
Also, by dividing Queries (Q) into multiple groups and having all Queries within a group share the same KV Cache, the overall memory footprint is significantly reduced.
Quantization: The unit is converted from 16-bit (FP16/BF16) to 8-bit (INT8) or even 4-bit (INT4). However, a "dequantization" process is required to back to its original format and higher compression ratios often lead to data loss. Recently, GPUs that support quantized unit operations have emerged.
Focus on Importance: Not all past data is necessary to generate the current response. Therefore, only the core parts with high importance are retained, while the rest are either deleted or compressed.

Representative examples applying these ideas include Google's TurboQuant and NVIDIA's KVCT.

TurboQuant

Traditionally, matrices are stored in Cartesian coordinates, but what if we changed the method to Radius and Angle? For example, the 2D Cartesian coordinate (x, y) of (1, 1) becomes (√2,45°) in Polar coordinates (magnitude, angle). The method of converting data into Angle and Magnitude—assigning 1 byte to the angle and 2 bytes to the magnitude. This allows for high compression of the angle data while maintaining precision in the more critical magnitude information.

Google reportedly reduced the cache size to up to 1/6 using TurboQuant. This is equivalent to compressing 16-bit FP16 data down to a 2.6-bit level. Even with such a reduction, overhead exists for dequantization and the processing of other functions, and it is expected that the actual compression level will be lower than this when maintained at an appropriate level of accuracy.

These compression techniques of TurboQuant lower data occupancy within fixed memory resources and ensure fast response speeds without latency even without a network connection, making them look most ideal for autonomous driving and mobile environments where real-time decision-making is critical.

KVCT (Key-Value Cache Transform Coding)

This approach introduces media compression principles to KV Cache compression, treating data as a signal with specific patterns and flows. By analyzing data volatility, it focuses on significant segments and allocates high precision to data that most heavily influences model performance.

While this method can compress data by up to 1/20, it is a form of lossy compression, meaning that accuracy may decrease due to information loss. Additionally, a notable disadvantage is that it is currently optimized specifically for NVIDIA hardware.

Solutions in Hardware and the impact on pricing

The main strategy is to bridge the gap between DRAM and SSD by increasing both speed and capacity through the following approaches:

A. Focus on Low Latency

XL-Flash: A product from Kioxia designed for ultra-low latency. By using SLC (Single-Level Cell) technology, it enhances processing speed and lifespan. It allows for small memory access in 512 bytes, which prevents fragmentation and enables the handling of small, quantized data pieces.
Z-NAND: Samsung's solution aimed at low power consumption and ultra-low latency through enhanced controller functions. Like XL-Flash, it is based on SLC. In practice, its read latency is approximately 12–20 µs, which is more than five times faster than the 100 µs latency of standard eSSDs.

B. Increasing Bandwidth

HBF (High Bandwidth Flash): Positioned as a middle tier between HBM and SSD. First-generation products are expected to stack 16 dies of 256 Gbit each to provide a total capacity of 512 GB. Notably, they are designed to offer nearly the same physical size, power consumption, and stack height as HBM4. SK Hynix and SanDisk are currently leading the global standardization of HBF, with HBF-based AI device samples expected to emerge in early 2027.

C. Modularization through Logic and Storage Integration

NVIDIA CMX (Context Memory eXtension) combines DPUs with high-performance SSDs (XL-Flash, Z-NAND, etc.). It allows multiple GPUs to access data simultaneously with lower power consumption and faster processing speeds than conventional methods. Vera-Rubin Platform features the inference-specific LPX (Low latency Performance eXtension), which prioritizes raw computation speed. LPX plans to connect 256 Groq3 LPUs—each equipped with 500MB of SRAM—to satisfy requirements for both high throughput and low latency.
AMD is collaborating with storage specialist VAST Data to build a platform similar to CMX.
Google is following a strategy similar to LPX with its 8th-generation TPU (TPU v8i). It has tripled the SRAM capacity to 384MB compared to the previous generation and minimized the connection paths between chips using a "Boardfly topology” to optimize performance.

The Trade Ratio Dilemma to supply Gen AI Memory

In a traditional semiconductor cycle, Moore's Law dictates that the number of transistors on a microchip doubles approximately every two years. In DRAM, this means the advanced technology nodes generate more chips per wafer leading the cost reduction which allows demand growth based on the cheaper pricing per bit. AI-driven HBM, high bandwidth memory, breaks this rule.

To achieve higher speeds and increased I/O, for higher bandwidth, logic blocks must expand, physically enlarging the chip size and reducing the number of net dies per wafer.

For instance, a 16Gb DDR5 chip can yield 1,310 dies in the 1a process, 1,680 in 1b, and 2,180 in 1c. While results may vary slightly based on yield, they do not differ significantly across suppliers.
In contrast, for HBM, the number of dies decreases from 580 to 500 when moving from HBM3e to HBM4, both based on a 24Gb capacity. HBM5 follows a similar trend, with the number of producible chips dropping by approximately 10% compared to the previous generation.
If commodity DDR5 and HBM3e were produced using the 1b process in 2025, the Trade Ratio (Net die of DDR5/HBM3e) would be 2.5x.
However, comparing the 1c process and HBM4 in 2026, this ratio expands to 3.7x (assuming a 90% yield for commodity and 70% for HBM).
As logic designs become more complex to achieve faster speeds and data reliability for inference, it is unlikely that this Trade Ratio will decrease. This situation will remain unchanged unless more efficient ASICs, surpassing GPUs, are deployed on a large scale.

(Please see the previous edition of Industry Beats on April 19 2026, Pressuring Issues in Supply Crunch)

High-Stakes Tech Translation in NAND

Similar to the case in HBM for DRAM, Efforts to create high-bandwidth and low-latency memory are also essential in NAND flash. To make naturally slow NAND faster, manufacturers need many more logic blocks. Furthermore, they are shifting from standard triple-level cell (TLC) or quad-level cell (QLC) structures to faster single-level cell (SLC) designs.

Developing next-generation NAND for generative AI requires larger chip sizes and three to four times the usual manufacturing capacity. High-bandwidth memory (HBM) already suffers from a fourfold loss in chip size and lower yields. Forcing a shift to SLC on top of that requires an additional three-to-fourfold loss in manufacturing capacity. This combination creates a fatal weakness in production costs.

Rising NAND Price. Since all KV Caches cannot be loaded into HBM or DRAM, there is a push to store past conversation histories or inactive caches in high-performance eSSDs. This has led to strong demand, driving up NAND prices.

In Q1 2026, the average price for enterprise NAND reached $0.28/GB, a 110% increase compared to Q4 2025. It is expected to rise by more than 50% again in Q2 2026, reaching $0.43/GB. As supply shifts toward servers, mobile and PC NAND prices are also rising sharply. The average price for mobile NAND, which was $0.18/GB in Q1 2026, is projected to rise 90% to $0.26/GB in Q2.

The rise in both DRAM and NAND prices is expected to cause an overall increase in electronics prices this year, potentially leading to a decrease in demand. This temporary dip in demand is expected to recover after 2028 when memory supply becomes more visible, and it is anticipated to hasten the activation of On-Device AI based on advanced software technologies.

Memory is integrated with logic and transition toward a custom memory. As memory performance and capacity within GPUs/ASICs become critical, HBM performance is emerging as a differentiating factor for chipsets. A prime example is NVIDIA's requirement for speeds of 11.7 Gbps, which exceeds JEDEC standards. Strong demand for HBM and server memory is expected to continue for the time being.

With the emergence of processors equipped with SRAM specifically for inference, the importance of logic is increasing. While a DRAM cell consists of 1 transistor and 1 capacitor and is sold as a standalone product in massive arrays, SRAM cell is logic composed of 6 transistors and is used inside the processor. To provide faster memory, memory manufacturers must pay attention to memory forms integrated with logic and transition toward a custom-ordered memory business.

Receive our insightful weekly newsletter and stay ahead of the competition.

Author

Jeongku Choi

Jeongku Choi is a Research Analyst at Counterpoint Research based in Seoul, Korea. As an engineer in LG Electronics, she was responsible for commercializing the latest technologies for new products in mobile and automotive. Also analyzing and strategizing the latest Android technology was one of her roles. Her interests are mobile, mobility, AI, and emerging technologies.