Reduce AI Inference Costs with Nvidia's Blackwell Platform

Reducing AI Inference Costs

Nvidia announced that its Blackwell platform enables four leading inference providers to cut costs per token by up to 10 times. This analysis, released on Thursday, highlights how enhancements in hardware and software contribute to this reduction.

The improvements were relevant for sectors such as healthcare, gaming, and customer service. A deployment study by Baseten, DeepInfra, Fireworks AI, and Together AI reveals how companies scale artificial intelligence (AI) from pilot projects to millions of users.

Optimization Model and Its Implications

According to the analysis, cost reduction depends on the joint use of Blackwell hardware, optimized software stacks, and the transition from proprietary models to open-source models. Simply improving the hardware resulted in gains of up to 2x, but a greater reduction requires adopting low-precision formats such as NVFP4.

Dion Harris, senior director of HPC and AI solutions at Nvidia, stated: "Performance drives the reduction in inference cost." This logic implies that increasing throughput, the ability to process more data concurrently, results in lower costs per token.

Successful Cases in Practice

Nvidia detailed four success cases that illustrate the combination of Blackwell infrastructure, optimized software stacks, and open-source models. One example is Sully.ai, which reduced inference costs in healthcare by 90% by transitioning to open-source models, saving millions of minutes of doctors' time.

Another case, Latitude, reported a 4x reduction in inference costs for its AI Dungeon platform, lowering the cost per million tokens from 20 cents (on the Hopper platform) to 5 cents after adopting NVFP4. This technical shift was crucial for cost optimization.

Reference to the Sentient Foundation shows an improvement of 25% to 50% in cost efficiency in its chat platform, thanks to the use of the optimized inference stack from Fireworks AI. This increase in efficiency is vital, especially when latency is a critical factor.

Technical Factors Influencing Cost Reduction

The range of reductions from 4x to 10x reflects different combinations of optimizations, with three main factors highlighted:

• Adoption of precision formats: NVFP4, for example, reduces the number of bits needed to represent model weights, allowing for higher computation per GPU cycle.
• Model architecture: Mixture of Experts (MoE) models leverage the fast communication provided by Blackwell's NVLink architecture, making them more efficient.
• Integration of software stacks: Nvidia's co-design approach facilitates hardware and software optimization, resulting in improved performance.

Necessary Assessment and Testing

Companies planning to migrate to inference with Blackwell should assess whether their workloads justify infrastructure changes. Shruti Koparkar from Nvidia suggests that companies must consider the volume of requests and the latency sensitivity of applications.

Testing with real production loads is essential. Koparkar notes that throughput metrics may not reflect real operational conditions. The phased approach used by Latitude can serve as a practical guide as companies evaluate cost and efficiency improvements.

Diversity of Providers and Economic Considerations

While Blackwell is a promising option, other platforms like AMD MI300 and Google TPU also offer alternatives. Solid evaluations should consider total costs, including operational overhead, and not just the cost per token, to determine the most economical approach.

With the inference market continuously evolving, companies must be ready to explore different sources, optimize their workflows, and ultimately adopt solutions that best meet their specific needs.