NVIDIA TensorRT LLM supports language and visual generation inference

NVIDIA’s TensorRT LLM optimizes inference for large language models and visual generation models on NVIDIA GPUs, including support for diffusion models. Built on PyTorch, the open-source library provides a high-level Python API for deployments spanning individual GPUs, multiple GPUs and multiple nodes. Developers can customize predefined models and extend the runtime using native PyTorch code.

Performance features include specialized kernels for attention, matrix multiplication and mixture-of-experts operations, alongside speculative decoding, prefill-decode disaggregation and expert parallelism. The framework integrates with NVIDIA Dynamo and Triton Inference Server. A collection of quantized and optimized models is also available through Hugging Face.

TensorRT LLM collects anonymous telemetry by default to help prioritize engineering work. Collected information includes hardware and software details, deployment configuration and feature usage. Prompts, outputs, model weights, model paths and user-identifying information are excluded. Developers can disable collection through environment variables, a configuration file, the Python API or command-line flags, and the telemetry implementation is open source and auditable.