NVIDIA emphasizes tokens per megawatt for AI factories

At AI Infra Summit, NVIDIA framed AI infrastructure around energy efficiency and tokens per megawatt as agentic workloads increase demand for performance, scale and power-aware operations. Ian Buck, vice president of hyperscale and high-performance computing, outlined collaborations with Amazon’s Annapurna Labs on NVHBM custom high-bandwidth memory and d-Matrix on NVLink Fusion, combining NVIDIA Vera CPUs with d-Matrix Raptor XPUs for low-latency inference.

NVIDIA positioned Vera Rubin systems, Dynamo inference software, NeMo libraries and networking technologies including NVLink, Spectrum-X Ethernet, ConnectX SuperNICs and BlueField as a full-stack AI factory platform. DSX MaxLPS can deliver up to 1.4x more tokens per megawatt through factory-wide power optimization, while Lambda reported 23% better performance per watt after running 19 nodes within a power budget typically used for 16 full-power nodes.

Emerald AI and NVIDIA demonstrated automated load reduction with Silicon Valley Power, with DSX Flex responding to demand signals while protecting priority AI workloads. Pinterest is using NVIDIA Blackwell and Dynamo for conversational AI in visual discovery, while Vera Rubin NVL72 results on SemiAnalysis AgentX showed up to 30x higher throughput per megawatt than NVIDIA GB300 NVL72 and up to 45x lower cost per million tokens.

Additional demonstrations highlighted Groq 3 LPX for deterministic ultralow-latency inference, startup benchmarks for the Vera CPU across agent and data workloads, and NVLink 6 resiliency features designed to keep very large GPU deployments operating reliably.