Google releases llm-d 1.0 for Kubernetes inference

Google and its partners have released llm-d 1.0, an open-source stack for distributed large language model inference on Kubernetes. Backed by more than 500 contributors from 25 organisations under the Cloud Native Computing Foundation sandbox, the project connects model-serving tools with production infrastructure. It targets agent workloads and reinforcement learning rollouts that can create traffic spikes, duplicate cached context and compete with interactive requests.

The stack combines cache-aware routing, tiered key-value storage and priority controls to distribute requests across vLLM and SGLang deployments and mixed TPU and GPU fleets. Stable APIs, higher-availability routing and upgrade tools aim to preserve active sessions during updates. Its DisaggregatedSet feature manages separate prefill and decode pools, allowing independent scaling, while an asynchronous processor handles queued jobs alongside real-time traffic. Integration with Google Kubernetes Engine adds managed routing and autoscaling support.

Google says time-slicing raised accelerator utilisation from about 40% to more than 70% in tested scenarios. In a coding benchmark on Ironwood TPU, prefix-aware routing and host-memory offloading reduced median time-to-first-token by more than 85% and increased throughput by 2.4 times compared with a standard Kubernetes Service. Snap says it is integrating the stack into production infrastructure, while Mistral supports its routing and priority controls for managing traffic spikes.