llm-d advances production AI inference

llm-d is positioning its inference control plane around workloads that stretch beyond conventional text serving. Recent updates emphasize multimodal pipelines, where image and media inputs change request size, cost, time to first output token, cache identity and processing stages. The platform addresses those shifts with a token estimator, content-hash routing, a retuned affinity gate and an encode tier, while vLLM remains the underlying model-server engine.

Production hardening is another theme. The v0.9 release adds high availability for the router, bounded flow-control defaults, a formal plugin lifecycle, KEDA-based autoscaling, DisaggregatedSet revision routing for P/D rolling updates, GPU utilization-aware endpoint scoring, diffusion model routing and hardware paths for GB200, GB300, Intel XPU, ROCm, and TPU v7. Observability work adds E2E distributed tracing across Gateway, EPP, KV-cache, P/D proxy and vLLM, plus fine-grained spans at routing and scheduling decision points.

Other updates focus on efficiency under load, including token-aware routing that balances KV-cache affinity against token load, peer-to-peer KV cache sharing when locality conflicts with load balance, heterogeneous vision-language serving across Intel and NVIDIA GPUs, and cooperative time-slicing for RL post-training on shared accelerators.