FreeToken targets local MoE inference beyond GPU memory

FreeToken is an AI inference engine built for high-speed local inference of large MoE models that exceed NVIDIA GPU VRAM. Rather than accelerating every LLM, it exploits the MoE pattern of activating only a subset of experts during inference, caching experts on the GPU and coordinating CPU-GPU execution when needed. The project is aimed mainly at NVIDIA GPUs such as the RTX 30 / 40 / 50 series, excluding DGX Spark.

Verified models include DeepSeek-V4, GLM-5.2 / GLM-4.7, Qwen3.6 / Qwen3.5 MoE, Qwen3-MoE, gpt-oss, Gemma-4, MiniMax-M2.5 and Muse-Glimmer, with support noted for other checkpoints using the same architectures and Dense models such as Qwen3.6-27B. Basic requirements include Linux x86_64, Python 3.10 or later, an NVIDIA GPU, drivers r580 or later and CUDA 13.

A local run used an NVIDIA GeForce RTX 3080 Laptop GPU 16GB (80W), RAM: 32GB, FreeToken: 0.1.2, PyTorch: 2.11.0 + CUDA 13.0 and nvidia/Qwen3.6-35B-A3B-NVFP4. The model size was approximately 23.5GB, larger than the available 16GB of VRAM, yet it generated a short answer through FreeToken’s OpenAI-compatible API.

Generation testing measured long-form output of 256 tokens after warm-up, with runs of 70.79 tok/s, 72.95 tok/s and 72.81 tok/s. The reported average was 72.18 tok/s, average TTFT was 3.09 seconds and /v1/stats showed 73.1 tok/s. The result indicates that a 23.5GB MoE model can run locally at approximately 70-73 tok/s in this specific setup.