NVIDIA targets memory bottlenecks with coordinated hardware and software

NVIDIA is addressing rising GPU memory pressure by designing processors, networking, storage and software together. Larger models, longer contexts and agentic workloads are increasing demand as memory supply tightens and prices rise. Adding high-bandwidth memory (HBM) alone cannot resolve the problem because bandwidth limits how quickly stored data reaches compute resources.

The approach reduces model weights and KV cache, which stores context for active requests, using NVFP4 precision. Distributing model weights across GPUs leaves more memory available for concurrent users. Vera CPU memory provides nearby capacity for paused context, while Dynamo inference software moves inactive context into lower-cost CMX storage and retrieves it before reuse. Dynamo also coordinates context reuse across GPU memory, CPU memory and storage.

For retrieval-augmented generation, Nemotron Retriever models create smaller embeddings, while the cuVS library supports building and serving search indexes outside HBM. Only retrieved passages needed by the model enter its context, reducing competition for expensive GPU memory.

NVIDIA reports that, on the SemiAnalysis AgentX DeepSeek V4 Pro workload at 125 tokens per second per user, Vera Rubin NVL72 delivers up to 30x higher throughput per dollar than GB300 NVL72, while HBM capacity per GPU increases about 3%. The comparison highlights its focus on useful output within power and response-time constraints rather than memory capacity alone.