SambaNova pairs RDUs with Nvidia GPUs to accelerate inference

Intel-backed SambaNova reported third-party benchmark results showing its heterogeneous inference platform outperforming GPU-only systems. Artificial Analysis found the company’s SN50-series accelerators reached 763 tokens a second in MiniMax M2.7 at short context lengths of 10,000 input tokens. SambaNova says the platform can sustain more than 450 tokens a second at longer context lengths.

The system splits inference work across different hardware. The prefill phase ran on four Nvidia H200 GPUs, while decode operations were handled by a SambaNova rack with 16 SN50 accelerators. SambaNova is positioning the approach as a way for customers to extend aging GPU fleets with air-cooled decode accelerators that can be deployed in existing datacenters.

The company plans to demonstrate larger configurations with 128 and eventually 256 accelerators to show high token generation rates at high throughput. The results follow SambaNova and Intel’s announcement that Vector Core Compute would be among the first to deploy the combined GPU + RDU offering with TogetherAI as its first large-scale customer. SambaNova also completed the first close of a $1 billion Series F funding round led by General Atlantic, giving the startup an $11 billion valuation.