Nvidia Rubin architecture advances inference efficiency from GPU to rack scale
Nvidia has detailed the Vera Rubin architecture, which targets agentic AI inference through microarchitectural refinements in memory management, Tensor Core throughput, and inter-GPU communication efficiency. The platform, slated for release later this year, introduces specific o
Nvidia’s upcoming Vera Rubin platform, scheduled for release later this year, is designed to address the shifting landscape of AI workloads, specifically the transition from frontier training runs to agentic AI inference at scale. According to Tom's Hardware, the architecture aims to optimize performance and efficiency from the individual GPU to the rack level. The full Vera Rubin NVL72 rack-scale system integrates 36 Vera CPUs alongside 72 Rubin GPUs. The design prioritizes the rapid delivery of generated tokens, focusing on low unit costs and high throughput to support the computational demands of agentic workflows.
At the silicon level, the Rubin GPU utilizes two compute dies on a single package, connected via the Nvidia High Bandwidth Interface. The chip features 224 Streaming Multiprocessors (SMs) and 896 Tensor Cores, supported by 288GB of HBM4 memory that provides 22 TB/s of memory bandwidth. As an inference-focused accelerator, the architecture is rated for 50 sparse PFLOPS of NVFP4 inference throughput. These specifications underscore a design strategy focused on maximizing utilization of compute resources, ensuring that the hardware remains efficient while handling the dense matrix operations required for modern neural network inference.
Nvidia has refined the Tensor Memory Accelerator (TMA) to better handle Mixture-of-Experts (MoE) model architectures, where only specific sub-networks are activated per token. Previously, Blackwell GPUs required maintaining separate MoE descriptors in memory for each expert, creating computational overhead as the number of experts increased. According to Tom's Hardware, the Rubin TMA now supports GPU kernels that maintain and update a single unified MoE descriptor directly in the TMA instruction at runtime. This optimization reduces the calculation of metadata and decreases the overhead required for data movement, effectively freeing up GPU cycles for inference calculations.
The architecture also enhances Tensor Core performance by doubling the work performed on the K dimension, which is the shared inner dimension of matrix multiplication. This change allows calculations that previously required four loop iterations on Blackwell to be completed in two on Rubin. Furthermore, Nvidia has boosted softmax throughput within the GPU SM’s Special Function Unit (SFU). While Rubin maintains the 2X speedup in FP32 exponential math seen in Blackwell Ultra, it doubles BF16/FP16 exponential calculations compared to that predecessor, resulting in a 4X throughput improvement for those lower-precision data types.
To improve GPU utilization, Rubin introduces finer-grained dependency resolution between kernels. On Blackwell, a producer kernel on a thread block could delay the execution of a consumer kernel until the entire batch was finished. Rubin allows consumer kernels to begin execution on individual thread blocks as soon as the producer kernel’s output becomes available. This reduction in kernel-to-kernel latency is designed to increase tokens per second per user, ensuring that compute resources are not left idle while waiting for dependent operations to complete across the GPU.
Finally, the NVL72 rack-scale system optimizes inter-GPU communication over the NVLink fabric. To reduce synchronization overhead, Rubin introduces a feature called counted writes. According to Tom's Hardware, this replaces the memory barrier and atomic operations used in Blackwell with a single write counter update on the receiving GPU. This change reduces the network traffic and coordination latency required to share model weights and key-value cache data between GPUs. By streamlining these communications, the architecture aims to minimize the time spent waiting for coordination, thereby improving overall compute utilization within the rack.