Memory Solutions for Gen AI Part 1: Changing Capabilities
Generative AI (Gen AI) applications require high-speed, high-bandwidth, and low-latency memory to process vast amounts of data in real time. Rapid data access becomes all the more critical when it comes to inference, where real time decision and prediction is required.
Capabilities Needed:
- DRAM with the traditional interface has limitations in bandwidth and latency, making technologies like HBM, which stacks DRAM using TSVs, a crucial solution to meet these performance demands. The challenges and solutions related to memory design, and the emerging trends in memory technology are shaping the future of high-performance computing and the competitive landscape.
- Going forward, advancements in packaging technologies like 3D-IC and/or CoWoS are going to be adopted across different sectors, such as smartphones or PCs. In smartphones, which have space and cost limitations, various methods will be attempted to reduce latency and energy consumption without increasing cost and space.
- It is still unknown what types of Gen AI models and applications will be prevalent in 2030, and how many. Therefore, it will be critical to support the advancement in architecture and build up the ecosystem to be able to respond to any changes.
‘Memory Solutions for Generative AI’ will be published as a comprehensive series of reports covering the following topics: 1) Changing capability; 2) Changing competitive dynamics; 3) Made in China memory; and 4) Gen AI in mobile environment.
Category
Industry
AI, Semiconductors
Service
Standard
Report Type
Report
Time Period
Other
Receive our insightful weekly newsletter and stay ahead of the competition.
Author
Brady Wang
Hi, I’m Brady Wang, a seasoned professional with over 20 years of experience in the high-tech industry, spanning semiconductor manufacturing, market intelligence, and strategic advisory roles. Currently, I serve as an analyst at Counterpoint Research, where I specialize in semiconductors with a focus on advanced applications such as automotive, server platforms, and cutting-edge process nodes. My core research centers on AI servers and their key components, including GPUs, custom accelerators, high-bandwidth memory (HBM), CPUs, and advanced packaging technologies. I also track the evolution of AI server architectures, interconnect technologies, and data center deployment trends. By combining deep technical knowledge with market insight, I help clients navigate the fast-changing AI infrastructure landscape and make strategic, data-driven decisions.
MS Hwang
MS Hwang is a research director at Counterpoint, specializing in memory semiconductor research. MS Hwang brings over 30 years of experience from Samsung Electronics and sell-side brokerage research roles including ABN AMRO, Goldman Sachs, Credit Suisse and Samsung Securities.