Apple scales local model inference across Mac Studio clusters

Apple’s latest MLX release enables Muse Spark Open to run across four Mac Studio computers with M5 Ultra, generating over 36 tokens per second. The open-source framework combines unified memory, GPU Neural Accelerators and RDMA over Thunderbolt to support large-model inference on local hardware. Smaller models, including Qwen 3.8 and Mistral Medium 3.5, fit on a single Mac Studio with 512GB of unified memory.

Apple’s benchmarks show M5 Ultra delivering time to first token up to 4.6× faster than M3 Ultra, alongside 1.6× faster generation. Muse Spark Open needs multiple machines: its 4-bit quantized model occupies about 493GB before accounting for intermediate computations. MLX supports pipeline parallelization, which divides model layers between computers, and tensor parallelization, which distributes computation within each layer. Pipeline execution largely preserves generation speed while expanding available memory; tensor parallelization can accelerate generation but requires frequent, low-latency communication.

Scaling introduces tradeoffs. For Muse Spark Open at large contexts, communication overhead can make tensor parallelization across two machines slower than pipelining for initial prompt processing, although generation still benefits. Adding more machines improves performance. Developers can also launch a distributed local server through MLX and connect tools such as OpenCode, enabling agentic coding workflows without sending model inference to an external service.