llm-d 1.0 targets production inference challenges
Contributors to llm-d have released version 1.0 to help organisations run large language model inference at scale while managing infrastructure costs and operational complexity. Founded by Red Hat, IBM, Google Cloud, CoreWeave and NVIDIA, the open-source project is now governed by the Cloud Native Computing Foundation. It supports multiple model servers, including vLLM and SGLang.
The release adds multi-tenant quality of service, zero-downtime operations, a stable configuration API and agent-aware serving. These capabilities target fleets handling both latency-sensitive requests and less predictable agentic tasks. Existing features include cache-aware routing, KV cache offloading and separation of prefill and decode processing. Managing repeatedly reused context can consume more resources than generating final outputs, putting pressure on memory, throughput and latency.
IBM Research reported that open frontier models combined with llm-d enabled quantum error-correction hardware research at 11.5 times lower inference cost than an all-proprietary set-up. It reported a prefix-cache hit rate of about 98% and a blended serving cost of roughly USD $0.11 per million tokens. These results illustrate potential savings rather than a universal cost guarantee.
Production users include DigitalOcean, which uses llm-d for serverless inference and managed agents, and JetBrains, which runs it behind its Junie Lite plan. Roblox highlighted support for cloud and on-premises GPUs without dependence on a single inference engine.