Tech firms turn to custom chips to cut inference costs

Microsoft plans to begin large-scale deployments of its Azure Maia systems in 2027, with Maia 300 expected to become the primary inference platform for its products and services. The chip targets long-context reasoning, memory-intensive processing and AI agent tasks, with planned uses including Copilot and GitHub Copilot.

Inference, the process of running trained models to complete tasks, creates recurring computing costs as AI services attract more users. Custom processors can match hardware to specific workloads, potentially lowering operating expenses and generating more tokens within existing power constraints. They also give companies greater control over supply and infrastructure planning. Savings are not guaranteed, however: development, manufacturing, memory and software costs all affect the economics.

Google has separated its latest TPU designs into training and inference systems, while continuing to offer Nvidia GPU instances. Amazon uses Trainium for much of its Bedrock inference workload. OpenAI has announced Jalapeño, a custom inference processor developed with Broadcom, with deployment planned across data-centre partners. The partners reported improved performance per watt in early testing.

The shift challenges Nvidia in parts of the market rather than making its GPUs redundant. Its hardware and CUDA software ecosystem remain important for training, experimentation and varied deployments. Cloud providers are increasingly combining specialised chips with general-purpose accelerators, choosing processors according to workload, performance and cost.