Chips and Semiconductors

SlipStream Technology: An Innovative Solution to Overcome the Latency Obstacle in High-Bandwidth Flash Memory for GPU Systems

Researchers from NVIDIA, Stanford University, and Georgia Tech are developing SlipStream technology to reduce latency in High-Bandwidth Flash (HBF) memory, enabling massive storage capacities for Large Language Models without sacrificing performance.

2026-11-16
3 min read
14 views
certi.news Editorial Team
SlipStream Technology: An Innovative Solution to Overcome the Latency Obstacle in High-Bandwidth Flash Memory for GPU Systems

Large Language Model (LLM) inference processes face an increasing challenge in the need for massive memory capacities to accommodate huge models and increasing batch sizes. While High-Bandwidth Memory (HBM) provides ultra-high speed, its capacity remains limited and expensive. Hence, the idea of using High-Bandwidth Flash (HBF) emerged as an alternative, but it hit the obstacle of high latency.

The Difficult Trade-off: Capacity vs. Speed

HBF technology relies on replacing DRAM chips in HBM memory with NAND flash chips, providing storage capacity up to 16 times greater with the same read bandwidth. However, this advancement comes at a heavy technical cost; read latency in HBF reaches approximately 20 microseconds, which is equivalent to 20 times the latency of traditional HBM. This difference leads to slowing down inference processes by a rate of up to 12.8 times when relying on direct data access from HBF.

How Does SlipStream Technology Address This Gap?

A joint research team from NVIDIA, Stanford University, and Georgia Tech presented a research paper titled "SlipStream," a technology based on software-hardware co-design to transform the prefetching process into a "producer-consumer" pipeline.

The mechanism of operation relies on two levels:

  • At the software level: The technology analyzes the repetitive order of weight-page requests in LLM kernels, then inserts a "staging warp" that re-executes this order using non-blocking STAGE instructions.
  • At the hardware level: The "staging warp" acts as a data producer, while the "work warps" act as consumers. The technical genius here lies in tracking 4 KB pages instead of traditional cache lines, which significantly reduces SRAM memory consumption.

What Changes Practically in Performance?

Tests conducted on seven dense models with capacities ranging from 16 to 800 GB showed the overwhelming superiority of SlipStream technology over current designs such as H3. While the H3 design required a massive cache of up to 320 MB to achieve only 43.1% of peak performance, SlipStream was able to achieve 92.4% of peak performance using a cache of only 40 MB.

More importantly, the memory size required in SlipStream does not depend on HBF latency or the type of workload, making it a flexible and scalable solution. The technology also proved its efficiency in supporting Mixture-of-Experts (MoE) models, achieving 91.2% of peak performance, which opens new horizons for running the largest AI models at a lower cost and with much larger storage capacities.

News source
Semiconductor Engineering
Open original source ↗
c
Author

certi.news Editorial Team

In the same category

You may also like

View all news