Netflix builds in-house LLM serving stack

Netflix is moving beyond typical API consumption with a custom LLM serving infrastructure built for its production environment. The company’s AI Platform and Inference teams designed the system to support model deployment and inference while prioritizing flexibility, performance, and integration with existing workflows.

The architecture uses NVIDIA Triton Inference Server as the backend compute engine and adopts vLLM as the preferred inference engine over TensorRT-LLM. Netflix chose vLLM for its ability to load custom model architectures without complex compilation, support custom decoding logic, improve debuggability, and align with tools familiar to ML practitioners.

The serving system runs through a unified JVM-based ML platform, with LLM inference delegated to a remote Model Scoring Service that uses Triton. Netflix also exposes an OpenAI-compatible HTTP API alongside an internal gRPC interface, helping developers move from hosted models to self-hosted deployments with minimal code changes.

Operational work focused on GPU-heavy deployments, cold-start latency, fragmented metrics, and constrained decoding at scale. Netflix uses Red-Black deployments for stable interfaces and Versioned deployments when interface changes are needed, while a shift from vLLM V0 to vLLM V1 enabled batch-level constrained decoding and a C++ implementation to remove a CPU bottleneck.