• The memory wall from the bandwidth and cost perspective has been one of the key constraints in running LLMs efficiently and natively on the device.
  • Google’s Gemma 4 changes this using advanced quantization techniques to reduce memory requirements and expand edge AI deployment.
  • Apple, with its latest AFM 3 Core Advanced, uses a sparse model architecture that leverages NAND flash to reduce DRAM dependency.
  • As token consumption continues to rise, a hybrid AI architecture is the way to go with its balanced on-device and cloud-based AI processing.


The industry is facing challenges posed by the ongoing memory crisis, as modern foundation models require substantial memory just to load before inference begins, while larger parameter models consume significantly higher memory capacity for running the model weights. Most devices run within fixed memory envelopes, and a model that demands more memory will create bottlenecks and reduce the device’s processing speed.