Amazon and Google target Nvidia’s software advantage

Amazon and Google are trying to lower the software barriers that keep AI developers tied to Nvidia’s chips. Their tools aim to run existing PyTorch workloads on alternative hardware with minimal code changes, challenging an advantage built around Nvidia’s CUDA platform, libraries and development tools. Switching hardware also requires engineering work to verify model accuracy and performance.

Amazon released TorchNeuron as a public beta on October 20, supporting its Trainium2 and Trainium3 chips. For supported code, developers can change the device setting rather than rewrite models, while retaining familiar PyTorch testing and debugging tools. Voice AI startup Cartesia reported moving its inference pipeline with minimal changes. Deepgram said Trainium processes about 60% more input audio seconds per dollar than its existing GPU baselines in the benchmarks it evaluated.

Google’s TorchTPU similarly connects PyTorch to its Tensor Processing Units, aiming to preserve developers’ core training code and existing tools. Microsoft is also developing custom Azure chips and has announced preview software tools with PyTorch integration.

Important limitations remain. TorchNeuron’s initial release does not support dynamic and symbolic shapes, which allow data dimensions to change during execution. Google also cautions that running code successfully on different hardware is only part of the transition: developers may still need to adjust model designs or write specialised code to get the best TPU performance.