NVDA 156.14 ▼4.12%GOOGL 283.72 ▼4.07%MSFT 517.99 ▼1.12%AMD 504.28 ▼5.58%INTC 55.47 ▼7.02%TSMC 373.45 ▼6.34%AMZN 257.75 ▼3.34%META 626.41 ▼2.38%AAPL 352.78 ▼0.98%PLTR 160.39 ▼4.52%
Markets at last close

Infrastructure

llm-d 1.0 targets production inference bottlenecks

·1 min read

Contributors to llm-d have released llm-d 1.0, adding multi-tenant quality of service, zero-downtime operations, a stable configuration API and agent-aware serving. Founded by Red Hat, IBM, Google Cloud, CoreWeave and NVIDIA, the project is now governed by the Cloud Native Computing Foundation. It supports distributed model inference through servers including vLLM and SGLang.

The release addresses operational challenges that emerge as enterprises move AI into production: balancing competing workloads, maintaining service during upgrades and controlling infrastructure costs. Agentic systems repeatedly reuse context across turns and parallel tasks, putting pressure on memory, latency and throughput. Existing capabilities include cache-aware routing, KV cache offloading and separate prefill and decode processing.

Launch examples highlight potential savings, although the results reflect specific deployments. IBM Research reported 11.5 times lower inference cost than an all-proprietary set-up for work exploring quantum error-correction hardware, with a prefix-cache hit rate of about 98%. A separate demonstration served up to 3,000 concurrent coding agents with zero preemptions, reaching more than 6.6 million output tokens per minute at peak.

Production users include DigitalOcean, which uses llm-d for serverless inference and managed agents, and JetBrains, which runs it behind its Junie Lite plan. Roblox cited its flexibility for serving workloads across cloud and on-premises GPUs without dependence on a single inference engine.

Originally reported by itbrief.co.nzRead the source →
Related coverage