vllm-mlx brings vLLM-style model serving to Apple Silicon

vllm-mlx is a vLLM-style inference server for Apple Silicon Macs that runs on MLX and Metal with unified memory. It exposes OpenAI-compatible and Anthropic-compatible endpoints from one process, including chat, completions, embeddings, reranking, responses and messages, while supporting Claude Code through the Anthropic API.

The server is designed for higher-throughput local model serving with continuous batching, paged KV cache, prefix caching, SSD-tiered cache and warm prompts. It supports structured output through JSON Schema, MCP tool calling with parsers for OpenAI, Anthropic, Gemini, Qwen, DeepSeek and Gemma formats, and reasoning extraction for Qwen3 and DeepSeek-R1 models.

Multimodal support covers text, images, video and audio, with examples for Qwen3-VL, Gemma, Pixtral, Llama vision, Whisper speech-to-text and native text-to-speech. Performance examples include Qwen3-0.6B-8bit at 417.9 tok/s, Llama-3.2-3B-Instruct-4bit at 205.6 tok/s and Qwen3-30B-A3B-4bit at 127.7 tok/s on M4 Max, 128 GB.

Installation is available through uv, pip or source, with optional audio extras. The project is Apache 2.0 licensed and includes documentation for APIs, model acquisition and conversion, Prometheus metrics, benchmarking and multi-model serving.