• The requirement of computing power, latency and higher bandwidth can be met with the combination of advanced packaging and wafer fabrication techniques.
• One of the integration techniques widely used is CoWoS (chip-on-wafer-on-substrate), a 2.5D IC integration technology that integrates logic computing and HBM chips.
• As AI becomes more pervasive across devices and industries, vertical players such as Apple, Tesla, Google and Meta are adding tons of AI capability in their offerings and would also demand integrated AI compute for their portfolios. CoWoS seems the best bet for now.

Advanced packaging offers a good lever to enhance overall chip performance beyond traditional geometric scaling on transistors, and extend Moore’s Law over the next decade. Advanced packaging is categorized as front-end 3D which stacks chips or wafers vertically and backend 2.5D CoWoS that interconnects dies horizontally via RDL (redistribution layer) or interposer.

HBM going mainstream with demand for AI applications

The requirement of computing power, latency and higher bandwidth can be met with the combination of advanced packaging and wafer fabrication techniques, especially for high-performance computing chips. AI accelerators used to train AI models in data centers require the highest memory bandwidth available. Unlike earlier systems, which were memory-constrained, current data center architectures use a variety of techniques to overcome memory bottlenecks. One of the solutions widely implemented to boost bandwidth and memory capacity is high bandwidth memory (HBM). AI has been a big driver of HBM in GPUs even though traditionally it has been a niche technique.

HBM technology works by vertically stacking DRAM chips on top of one another and interconnected through TSVs (Through Silicon Vias) and micro-bumps. For AI training and high-performance applications, HBM can deliver terabytes per second which is significant processing power required for AI and high-performance computing applications as the DRAMs are stacked. While HBM offers extreme bandwidth for the off-chip memory needed for data center AI accelerators, tradeoffs include HBM’s cost and thermal limitations. HBM is becoming more mainstream, with advances in the area, performance improvement and reduced power consumption with every iteration, as the DRAM stack and the SoC are placed in a single package substrate, driving its adoption in AI applications.

CoWoS major driver for packaging advanced logic with HBMs

One of the integration techniques widely used is CoWoS (chip-on-wafer-on-substrate), a 2.5D IC integration technology that integrates logic computing and HBM chips by mounting them on a silicon interposer and then placing them directly on a package substrate. TSV/RDL interposers are for extremely fine-pitch, high-I/O, high-performance and high-density semiconductor IC applications.

The logic and HBMs are first bonded side by side on the silicon interposer to form chip-on-wafer (CoW) with fine pitch and high-density interconnect routing among the devices. Each HBM consists of DRAMs with micro-bumps and a logic base with TSVs straight through them. Through-Silicon Via (TSV) is the feature that enables 2.5D and 3D advanced packaging. TSVs are electrical connection pathways that are short vertical columns running through the silicon wafer or die and enable smaller package sizes and denser interconnects, improve electrical performance by shortening the electrical distance traveled, and enable stacking of multiple chips used in products like HBM. Finally, the TSV interposer’s assembly with larger bumps on a package substrate is done.

Source: Counterpoint Research

CoWoS also a bottleneck for industry

Over the years, CoWoS technology development has focused on supporting increasing silicon interposer dimensions to support processors and HBM stacks in the overall package. Today, CoW with C2 bump by TCB method is the most used assembly method for silicon-to-silicon flip chip bonding. CoW flip chip by a bumpless technique called the hybrid bonding method, currently in R&D, will gain traction over the years.

CoWoS has been the main bottleneck for AI shipments recently. Increasing CoWoS capacity to fulfill the rising demand, driven by NVIDIA, AMD, Broadcom and Amazon, has been the focus.

Addressing challenges to CoWoS growth

To address these challenges, existing players need to diversify the supply chain by increasing capacity and collaboration with players in the value chain – OSATs, interposer manufacturers, and packaging equipment vendors.

What should pure-play foundries do?

TSMC has capacity constraints with respect to CoWoS creating some bottlenecks for NVIDIA and other customers. One of the reasons here is the bottleneck from equipment suppliers and manufacturing interposers. Therefore, TSMC should look to collaborate and potentially outsource to its partners, which will help stabilize volume in the near- to mid-term and meet the growing demand for advanced packaging in AI and HPC applications. So, long term, TSMC should collaboratively work with equipment manufacturers to optimize the CoWoS processing techniques such as TCB (Thermal Compression Bonding) and hybrid bonding.

What should equipment vendors do?

Equipment players should collaborate closely with foundries and co-develop optimized processes to scale the CoWoS packaging. For example, Applied Materials is actively developing new technologies for hybrid bonding and TSV to advance heterogeneous chip integration that help chipmakers integrate chiplets into advanced 2.5D and 3D packages. Similar collaborations would be expected from other players such as KLA Tencor, Lam Research, ASMPT and BESI to drive advancements in CoWoS packaging or other competing techniques such as hybrid bonding techniques to enable improvements in fine pitch, I/O density and power consumption. Research collaborations will provide semiconductor and systems companies with a complete suite of tools and technologies for developing and prototyping various packaging designs and enable continued advances in power, performance, area, cost and time-to-market.

What should IDMs do?

Samsung and other IDMs are also actively looking to take a share of the advanced packaging market and enhance their design and manufacturing capabilities to establish a sufficient customer base for high-end solutions. For example, Samsung Electronics’ HBM-PIM technology integrates processing chips and advanced memory chips like HBM with its proprietary 2.5D or 3D packaging technology. Samsung has developed HBM-PIM as a potential replacement for its existing HBM2 product where the bottom four of eight memory chips are replaced with chips containing both DRAM and compute cores. By doing some of the compute in the DRAM, the volume of data that needs to go to the processor decreases, effectively speeding up the neural network and saving the power needed to transport data. As this technology develops, Samsung will be able to leverage its advanced packaging solutions and technology as it is the only company that has a chip business portfolio spanning memory chips to contract-based processing chip fabrication and packaging and will be able to increase its market share and compete with TSMC.

Advanced packaging is also an opportunity for IDMs that have mature node capability to enhance the performance of their existing product portfolios and capture significant market share.

What should OSATs do?

OSATs like ASE and Amkor are actively investing to expand advanced packaging offerings but it will take time to benefit from the uptrend. Yield and volume output are the keys for foundries to grow their advanced packaging platform. This also poses a barrier for most traditional OSAT companies to enter this area. However, they could collaborate with leading players and increase their capacity in back-end processes. Another option could be to increase investment in R&D for advanced packaging applications in mature nodes, which will help prepare for the next upturn, like moving away from flip chip and wire bonding.

OSATs will face rough weather if they don’t join the wave of advanced packaging to achieve PPA benefits in mature nodes. But they can get a first-mover advantage if they start investing now, especially in AI applications in the IoT, automotive and network segments.

What should fabless vendors do?

NVIDIA accounts for the majority of the current CoWoS capacity. Getting the high performance required for AI workloads is possible because of the benefits being derived from CoWoS packaging. The technology offers flexibility in terms of a wide range of interposer sizes, package sizes and a number of HBM cubes, and provides the best performance with the highest integration density required for high-performance computing applications.

As AI becomes more pervasive across devices and industries, vertical players such as Apple, Tesla, Google and Meta are adding tons of AI capability in their offerings and would also demand integrated AI compute for their portfolios. CoWoS seems the best bet for now. As other competitors including Intel expand their offerings to capture the AI demand, they would have to match the best in class, which as of now is provided through CoWoS.

From the client compute perspective, as the industry develops scaled-down AI models capable of running on smartphones and users demand AI models on smartphones, it might experiment with CoWoS. However, implementing CoWoS packaging in smartphone applications at present will not be cost-effective and alternative packaging techniques will be a better option to consider.

Related links:

Artificial Intelligence: Irrational Exuberance is in Full Swing

ABF Substrate Demand Likely to Recover in H2 2023, BT Substrate Demand to Remain Weak

Currency Fluctuation Limits Global Wafer Fab Equipment Revenue Growth to 9% YoY in 2022

Wafer Fab Equipment Revenue Tracker

Applied Materials FY 2022 Earnings Report

Applied Materials Delivers Strong FY 2022 Numbers Despite Headwinds

Data Center CPU Market: AMD Surpasses Intel in Share Growth

Global Data Center CPU Revenue Tracker: Q1 2018 – Q4 2022