Mesh LLM pools GPUs for local model inference

Mesh LLM launched July 11, 2026 as an open-source distributed inference engine for pooling GPU resources across multiple machines. The system exposes a single OpenAI-compatible API at localhost:9337/v1, uses iroh’s QUIC-based P2P networking for peer discovery and secure transfer, and ships as an 18MB static binary.

The engine splits dense transformer models with pipeline parallelism and Mixture of Experts models with expert parallelism, assigning layers or experts based on available VRAM and bandwidth. It supports Hugging Face and GGUF models, includes speculative decoding, and advertises support for models up to 235B parameters. Nodes can discover peers through iroh gossip without a central registry, with NAT traversal available for machines outside the same network.

The main appeal is cost control and hardware utilization. The source compares 500 million tokens per day on GPT-4o-class models at $225,000 per month with approximately $7,200 per month for pooled GPU rental, while noting that idle GPUs often sit unused across cloud instances, on-prem servers, and developer workstations. Limitations include WAN latency, required GPU runtimes such as NVIDIA CUDA 12.x or AMD ROCm 5.x, and the instability expected from a v0.1.0 project.