llm-d 1.0 adds production controls for distributed inference

Contributors to llm-d have released llm-d 1.0, strengthening the open source platform for running distributed large language model inference in production. Governed by the Cloud Native Computing Foundation, the project helps organizations manage serving costs, hardware utilization and the demands of context-heavy agentic workloads. Its community includes more than 500 contributors from over 25 organizations.

The release adds priority and fairness-aware traffic controls so latency-sensitive requests and bursty agent workloads can share infrastructure. High-availability routing, rolling updates and coordinated upgrades support maintenance without interrupting traffic. Session-aware routing keeps conversations near cached context, while scheduling and Responses API state support help agents reuse context and reduce recomputation. A stable configuration API and support across clouds, accelerators and model servers provide a common operational foundation.

In a benchmark demonstration, researchers reported over 6.6 million output tokens per minute at peak while serving up to 3,000 concurrent coding agents with zero preemptions. At current cloud rental rates, the deployment’s per-token costs would have been 5 to 10 times less than equivalent commercial API pricing, with the largest savings on input-heavy workloads. Google Cloud is integrating the release into Google Kubernetes Engine across its Google TPU and NVIDIA GPU portfolio. DigitalOcean and JetBrains also report using llm-d in production services.