Integrated GPUs make local AI practical with Vulkan offload

Vulkan offload in llama.cpp lets local AI workloads move tensor operations from the CPU to integrated graphics without CUDA, ROCm or an Nvidia card. Intel Iris Xe, AMD Radeon graphics in Ryzen chips and Apple Silicon through comparable GPU offload approaches can use shared memory to accelerate quantized model inference on ordinary laptops.

Performance varies by chip, model size and memory bandwidth. CPU-only inference on a modern eight core laptop with a 7B parameter quantized model usually lands somewhere between three and seven tokens per second, while Iris Xe or Radeon 780M with Vulkan offload typically reaches the eight to fifteen tokens per second range. Iris Xe can deliver eight to twelve tokens per second on prompt evaluation and six to ten tokens per second on generation for Q4 7B models, while Radeon 780M reaches twelve to twenty tokens per second on Q4 7B models and thirty to fifty tokens per second on 3B models. A base M2 with 16 GB of unified memory runs Q4 7B models at thirty to forty-five tokens per second using Metal.

The practical ceiling remains clear. 13B class models are borderline, 30B and up effectively requires discrete VRAM, and anything past 16,000 tokens strains shared memory bandwidth. Integrated graphics are enough for learning, prototyping, personal tools and small local assistants, but larger models, long contexts and multi-user serving still point to discrete GPUs or cloud inference.