NVDA 125.53 ▼14.01%GOOGL 265.29 ▼2.70%MSFT 484.52 ▼2.77%AMD 424.39 ▼13.25%INTC 51.28 ▼7.97%TSMC 318.71 ▼9.48%AMZN 246.82 ▼4.10%META 560.84 ▼5.93%AAPL 342.56 ▼1.90%PLTR 143.49 ▼6.63%
Markets at last close

Infrastructure

llm-d 1.0 targets production inference challenges

·1 min read

Contributors to llm-d have released version 1.0 to help organisations run large language model inference at scale while managing infrastructure costs and operational complexity. Founded by Red Hat, IBM, Google Cloud, CoreWeave and NVIDIA, the open-source project is now governed by the Cloud Native Computing Foundation. It supports multiple model servers, including vLLM and SGLang.

The release adds multi-tenant quality of service, zero-downtime operations, a stable configuration API and agent-aware serving. These capabilities target fleets handling both latency-sensitive requests and less predictable agentic tasks. Existing features include cache-aware routing, KV cache offloading and separation of prefill and decode processing. Managing repeatedly reused context can consume more resources than generating final outputs, putting pressure on memory, throughput and latency.

IBM Research reported that open frontier models combined with llm-d enabled quantum error-correction hardware research at 11.5 times lower inference cost than an all-proprietary set-up. It reported a prefix-cache hit rate of about 98% and a blended serving cost of roughly USD $0.11 per million tokens. These results illustrate potential savings rather than a universal cost guarantee.

Production users include DigitalOcean, which uses llm-d for serverless inference and managed agents, and JetBrains, which runs it behind its Junie Lite plan. Roblox highlighted support for cloud and on-premises GPUs without dependence on a single inference engine.

Originally reported by itbrief.asiaRead the source →
Related coverage