vLLM now runs on Vera Rubin NVL72: 7.8x higher throughput per GPU

vLLM now offers full support for Vera Rubin NVL72, NVIDIA's new generation of GPUs designed for large-scale agentic inference. The work is the result of close collaboration between the vLLM Team, Inferact, Red Hat, and NVIDIA. The results are impressive: 7.8x higher throughput per GPU compared to GB200 NVL72 on the AgentX benchmark, and 3.7x higher throughput on MLPerf Inference v6.1's VLM benchmarks.

Architectural advantages of Vera Rubin

Vera Rubin NVL72 brings substantial improvements. The platform delivers 5x more NVFP4 FLOPS for low-precision inference — critical for fast inference on large models. Memory has been upgraded to HBM4, offering 2.4x higher bandwidth than the HBM3e used on GB200. Chip-to-chip interconnect has been improved with 6th-generation NVLink, delivering 1.7x higher bandwidth.

A key improvement for transformers involves the exponential operations used in attention mechanisms. Vera Rubin achieves 2x higher throughput for FP32 exponentials and 4x for BF16/FP16, allowing softmax operations to keep pace with faster GEMM operations. This matters because softmax is a bottleneck during decoding.

vLLM support and compatible models

vLLM provides day-0 support for Vera Rubin thanks to its architectural compatibility with Blackwell. A new compilation target, sm107, supports kernels compiled for sm100f without modification. Users can access this via nightly containers running CUDA 13.4 and PyTorch 2.15 (the vllm/vllm-openai:cu134-nightly image).

Supported models include DeepSeek, Moonshot AI (Kimi), Z.ai, MiniMax, and GLM. All of them work out of the box thanks to kernel compatibility with the Blackwell architecture.

Optimizations for Vera Rubin

vLLM leverages locality domains from CUDA 13.4 to optimize MoE layers. Weights are split across NUMA memory domains, allowing each SM to read data only from the nearest memory. With MiniMax M3, this delivers a 1.2x speedup on small decoding batches.

Through FlashInfer 0.7.0, specialized kernels have been integrated: NVFP4 and MXFP4 GEMM for low-precision matrix operations, NVFP4 MoE for efficient expert routing, FP8 attention, and FP8 MSA prefill for faster prefix caching.

Real-world performance

On SemiAnalysis's AgentX benchmark (MiniMax M3): vLLM achieves 7.84x higher throughput per GPU while maintaining interactivity, and 5.18x under a 150 TPS cap.

On MLPerf Inference v6.1: the Qwen3-VL-235B model (235 billion parameters), running vLLM as the backend, delivers 3.7x higher throughput than on GB300 NVL72 across all three scenarios — offline, server, and interactive.

Roadmap

The vLLM community continues to optimize the platform. Upcoming priorities include: sm107 FlashInfer MegaMoE integration for large-scale models, full locality domain support across all MoE layers, overlapping kernels via PDL and Lamport Sync to hide latency, mega kernels for low-latency agentic scenarios, optimization of KDA and MLA kernels for Kimi K3, integration of CSA and HCA kernels for DeepSeek-V4.1-Flash, completion of Rubin MSA decode kernels, and integration of the CFT counted-write MoE all-to-all kernel.

The work is being carried out jointly with NVIDIA, Inferact, and Red Hat, with active support from the vLLM community.