llm-d 1.0 adds production controls for distributed inference

Contributors to llm-d have released llm-d 1.0, adding operational capabilities for organizations serving large language models at scale. Governed by the Cloud Native Computing Foundation, the project combines distributed inference tools with controls designed to improve reliability, hardware utilization and serving costs. Its architecture supports multiple model servers, including vLLM and SGLang, alongside cache-aware routing, cache offloading and separate prefill and decode processing.

The release introduces priority and fairness-aware flow control so latency-sensitive requests and bursty agent workloads can share infrastructure. Router high availability, hot standby and coordinated rolling updates are designed to keep traffic flowing during maintenance. A stable configuration API provides a foundation for deployment, while session-aware routing, turn-priority scheduling and native Responses API state support help agents reuse cached context and reduce recomputation.

Reported deployments and benchmarks illustrate potential efficiency gains. IBM Research used open frontier models and llm-d to explore quantum error correction hardware at 11.5 times lower inference cost than an all-proprietary configuration, reporting a prefix-cache hit rate of approximately 98% and a blended serving cost of about $0.11 per million tokens. A separate agentic benchmark deployment reached over 6.6 million output tokens per minute at peak, serving up to 3,000 concurrent coding agents with zero preemptions. Google Cloud is integrating the release into Google Kubernetes Engine across its Google TPU and NVIDIA GPU portfolio.