Configuring and optimizing Kubernetes for new GPU hardware such as NVIDIA Vera Rubin is complex and error-prone. Drivers, kernels, runtimes, networking, operators, and schedulers must work together, with settings tuned to the hardware and workload. Small mismatches can prevent workloads from running or limit performance.
The NVIDIA AI Cluster Runtime (AICR)—part of the NVIDIA DSX AI Factory Platform—simplifies this work by capturing tuned configurations in reusable recipes, generating bundles for existing deployment tools, and validating the running cluster. It will help Kubernetes environments be ready for NVIDIA Vera Rubin NVL72 compute trays on Day 1.
Overall, the NVIDIA DSX AI Factory Platform brings together technologies to help the ecosystem design, build, and operate more efficient AI factories, including open source, modular software components purpose-built to support multi-tenant operations and scaling.
Configuring new hardware architecture in a cloud-native stack requires plumbing updates from the lowest-level bindings all the way up to the orchestration layer. It is still early days for Vera Rubin.
This post covers what changes if you run NVIDIA GB200 NVL72 today, what has been validated on early hardware access using NVIDIA GPU Operator and the Dynamic Resource Allocation (DRA) driver with Vera Rubin support. It also shows how AICR brings these components together in a configuration that platform teams can reproduce and verify.
What changes from GB200 NVL72: planning your upgrade
If you run GB200 NVL72 today, NVIDIA Vera Rubin NVL72 is an evolution of the same NVIDIA MGX rack-scale architecture, now in its third generation. But several user-visible pieces change. Knowing about them now lets you plan cluster software and configuration changes alongside upcoming releases:
GPU driver: Vera Rubin NVL72 requires a new driver branch. It has been validated with R615.23, the host-installed R615.23 open GPU driver. Early support is targeted for R610, with official platform support in R620. AICR’s GB200 baseline is R580.173.02. R615.23 is the current tested version, not a confirmed minimum for official Vera Rubin support.
DOCA and NIC drivers: Vera Rubin NVL72 replaces the GB200 NVL72 reference design’s NVIDIA ConnectX-7 NICs and NVIDIA BlueField-3 data processing units (DPUs) with NVIDIA ConnectX-9 SuperNICs and NVIDIA BlueField-4 DPUs. DOCA 3.3.0 is the first GA release supporting ConnectX-9. The Vera Rubin system image validated on ships DOCA-OFED 3.3.0. This is the testbed version, not a requirement or a recommendation. AICR uses host-installed NIC drivers and DRANET for pod allocation here. Network Operator compatibility remains a separate qualification step (See the DOCA 3.3.0 release notes).
Device topology: The NIC/GPU/Direct Memory Access (DMA) topology of the Vera Rubin NVL72 compute tray differs from GB200, so GPU-to-NIC affinity and NUMA placement must be checked for I/O-intensive workloads such as Remote Direct Memory Access (RDMA) networking and GPUDirect Storage. Both reference trays contain two CPUs and four GPUs, but the Vera Rubin NVL72 trays expose eight ConnectX-9 PCI device groups with 32 functions, compared with four ConnectX-7 adapters on the NVIDIA DGX GB200 NVL72 compute tray. In both GB200 NVL72 and Vera Rubin NVL72 trays, GPUs 0 to 1 have CPU NUMA affinity 0; GPUs 2 to 3 have affinity 1. GPU-to-NIC connectivity is NODE within a socket and SYS across sockets. PCI function counts do not represent physical port counts (See the GB200 reference architecture).
IMEX: As on GB200 NVL72, multi-node NVIDIA NVLink workloads use IMEX. With DRA-managed IMEX, stop, disable, and mask the host’s static nvidia-imex service to prevent ownership conflicts. DRA v0.5.0 adds an alpha host-managed option with domain isolation only; our cluster uses driverManaged mode. (See the release notes)
NVIDIA Multi-Instance GPU (MIG) profiles: Rubin GPUs introduce new MIG profiles. R615.23 advertises these profiles on node. The AICR Vera Rubin recipes use whole-GPU allocation and do not configure MIG. MIG on Vera Rubin is not covered by this validation.
AICR recipes: All of these deltas are encoded in the new Vera Rubin recipe family, so aicr recipe and aicr validate reflect the validated combination rather than leaving you to rediscover each change (details are in the following sections).
Validation on Vera Rubin NVL72
To validate Vera Rubin NVL72 support end to end, we deployed the AICR runtime bundle on Vera Rubin clusters using the official GPU Operator v26.7.0 release and the official DRA driver v0.5.0 release, with a host-installed R615 driver.
Both the training and inference recipe variants now validate end to end for deployment and conformance. For training, an initial performance gate: a cross-node NCCL all-reduce over the NVIDIA NVLink fabric, running through an IMEX ComputeDomain and gated on a bus-bandwidth floor. GPU pods run cleanly, and IMEX channels are injected into claiming pods across multiple nodes, shown with real cluster output below.
A note on early-silicon realities: Bringing up new silicon is a process. The first validation rounds were temporarily blocked by a hardware/fabric issue in one of the early Vera Rubin clusters—unrelated to the GPU Operator or the DRA driver—that kept one compute tray out of the NVLink fabric. After the hardware was reseated, all GPUs on both nodes joined a single clean fabric clique.
From bring-up to reproducible recipes
Validating the components is only half the story. Standing up a working Vera Rubin cluster means choosing driver mode, container-runtime wiring, component versions, and a handful of platform-specific settings. AICR captures those choices as a versioned recipe, so the configuration on real Vera Rubin hardware is explicit, reproducible, and verifiable across clusters.
The new Vera Rubin recipe family (service: rke2; accelerator: Vera Rubin NVL72 (vr200); operating system: Ubuntu; with training, inference, and NVIDIA Dynamo inference variants) ships as Preview. It’s an early-adopter path that does not yet carry the production support and lifecycle qualification of a Supported recipe while the platform itself is pre-release. It encodes the decisions made during this bring-up:
Host-managed driver: The Vera Rubin system image pre-installs the arm64 open-kernel-module driver, so the recipe runs the GPU Operator with driver.enabled: false, while the container toolkit stays operator-managed. The recipe moves to the operator-managed driver when the official builds ship.
DRA driver with ComputeDomains: IMEX-backed ComputeDomains are enabled for the NVL72 fabric, with whole-GPU allocation remaining on the device plugin.
Platform floors and quirks: They’re encoded once so users don’t rediscover them. Kubernetes >= 1.34.1 — 1.34 for the GA resource.k8s.io/v1 DRA API that ComputeDomains/IMEX use, and the .1 patch because the RKE2 CDI/NRI toolkit path needs containerd 2.1 (v1.34.1+rke2r1). Ubuntu 26.04 with the Vera-optimized arm64 64k-page kernel. And smaller sharp edges, such as raising the NFD topology updater’s default 60Mi memory limit, which out of memory (OOM) kills on multi-terabyte NVL72 workers.
The first deployment of this bundle targeted a platform combination none of these components had ever been validated on together—arm64, Ubuntu 26.04, a 64k-page kernel, RKE2, and Vera Rubin—and went from cluster handoff to a healthy runtime in hours. A second Vera Rubin cluster reproduced the result the following day, and that repeatability is the point and benefit of the recipe.
The workflow on a live Vera Rubin cluster
To make this concrete, here is the actual AICR workflow, executed on a live Vera Rubin NVL72 reference cluster with the AICR v1.0.0 toolchain.
1. Generate the recipe
Recipe generation is offline: It resolves criteria against the recipe catalog and takes seconds.
$ aicr recipe \
--service rke2 \
--accelerator vr200 \
--os ubuntu \
--intent training \
-o recipe.yaml
[cli] building recipe from criteria: criteria=criteria(service=rke2,
accelerator=vr200, intent=training, os=ubuntu)
[cli] recipe generation completed: output=recipe.yaml components=14
componentNames=cert-manager, dra-node-labeler, dranet, gpu-operator,
k8s-ephemeral-storage-metrics, kai-scheduler, kube-prometheus-stack, nfd,
nodewright-customizations, nodewright-operator, nvidia-dra-driver-gpu,
nvsentinel, prometheus-adapter, prometheus-operator-crds
overlays=5
The resolved recipe layers five overlays into 14 components. Alongside the GPU Operator (host-managed driver mode) and the DRA driver (ComputeDomains enabled), it brings up the two pieces that are most specific to this platform: DraNet, the DRA network driver that allocates ConnectX-9 devices to pods (RDMA data-path performance over ConnectX-9 is not part of this validation), and exclusive RDMA netns mode, applied through NodeWright tuning. Each pod sees only its own Host Channel Adapter (HCA).
AICR also brings together components of the NVIDIA DSX AI Factory Platform: NodeWright applies node tuning through its native vr200/rke2 profile, KAI Scheduler provides workload scheduling, and NVSentinel monitors runtime health. Node Feature Discovery (NFD) and the Prometheus stack complete the recipe, with component versions pinned to the configuration validated on Vera Rubin hardware.
2. Render the deployment bundle
The bundle is a set of ordered, per-component Helm charts with a single deploy script:
$ aicr bundle -r recipe.yaml -o ./bundle
Files generated: 61
$ ls bundle
001-cert-manager 008-gpu-operator checksums.txt
002-dranet 009-k8s-ephemeral-storage-... deploy.sh
003-nfd 010-kai-scheduler README.md
004-nodewright-operator 011-nvidia-dra-driver-gpu recipe.yaml
005-nodewright-customizations 012-nvsentinel
006-prometheus-operator-crds 013-prometheus-adapter
007-kube-prometheus-stack
3. Snapshot and validate
AICR captures the cluster’s actual state, checks the recipe’s readiness constraints against it, and then runs its validation checks as in-cluster jobs:
$ aicr snapshot --output snapshot.yaml
$ aicr validate -r recipe.yaml -s snapshot.yaml --phase deployment
[cli] readiness constraint passed: name=OS.release.VERSION_ID
expected=26.04 actual=26.04
[cli] readiness constraint passed: name=OS.sysctl./proc/sys/kernel/osrelease
expected=>= 7.0 actual=7.0.0-2008-nvidia-bos-64k
[cli] phase completed: phase=deployment status=passed validators=4 passed=4
$ aicr validate -r recipe.yaml -s snapshot.yaml --phase conformance
[cli] phase completed: phase=conformance status=passed validators=8 passed=8
$ aicr validate -r recipe.yaml -s snapshot.yaml --phase performance
[cli] phase completed: phase=performance status=passed validators=1 passed=1
The performance phase runs the NVIDIA Collective Communications Library (NCCL) all-reduce gate as an in-cluster job: a cross-node all-reduce over the NVLink fabric through an IMEX ComputeDomain, checked against the recipe’s bus-bandwidth floor. The rke2/vr200 combination is registered in the validator’s applicability matrix and its runtime ships as an embedded template, so the validator provisions the ComputeDomain and the IMEX ResourceClaimTemplate itself—no operator prerequisites and no recipe-supplied runtime. It runs from a stock v1.0.0 release.
The full matrix now passes.
Phase | Check | Result | Disposition deployment | operator-health | pass | All components are healthy. deployment | gpu-operator-version | pass | Version gate met. deployment | expected-resources | pass | All expected Kubernetes resources are present and healthy. deployment | check-nvidia-smi | pass | nvidia-smi healthy on all nodes conformance | platform-health | pass | All platform checks passing. conformance | gpu-operator-health | pass | All operator components are healthy. conformance | accelerator-metrics | pass | DCGM metrics path verified end to end, all GPUs reporting. conformance | ai-service-metrics | pass | Custom-metrics API serving. conformance | dra-support | pass | DRA/ComputeDomain support. conformance | gang-scheduling | pass | KAI gang scheduling. conformance | pod-autoscaling | pass | External-metrics API plus HPA scale-up (1->3) and scale-down (3->1). conformance | secure-accelerator-access | pass | Device-plugin mode (no gpu.nvidia.com DeviceClass) performance | nccl-all-reduce-bw-nvls | pass | First cross-node NCCL all-reduce across both compute trays over multi-node NVLink (IMEX ComputeDomain): 744.71 GB/s busbw at 16 GiB.
The inference variant (Dynamo platform, inference gateway, gang scheduling, DCGM metrics) passes the same deployment and conformance matrix on the identical cluster.Two artifacts from that run are worth showing. First, the nvidia-smi banner on the pre-release driver is the exact change that tripped the validator on first contact. It’s a good example of the small surprises early silicon brings. The fix (NVIDIA/aicr#1748) is merged and verified against NVIDIA Vera Rubin hardware.
$ nvidia-smi | head -3
+--------------------------------------------------------------------+
| NVIDIA-SMI 615.23 KMD Version: 615.23 CUDA UMD Version: 13.4 |
+--------------------------------------------------------------------+
Second, live GPU telemetry flows through the standard DCGM exporter path. Note the placeholder product name, which resolves once PCI IDs are allocated near general availability:
DCGM_FI_DEV_SM_CLOCK{gpu="0",modelName="NVIDIA Graphics Device",
DCGM_FI_DRIVER_VERSION="615.23"} 120
4. The cross-node ComputeDomain test
Finally, the DRA/IMEX path end to end: a two-node ComputeDomain whose IMEX channels are claimed by pods on each compute tray. There’s one prerequisite on images with host-managed IMEX: The host’s static nvidia-imex service must be disabled and masked, since host-managed IMEX and DRA-managed ComputeDomains are mutually exclusive (DRA driver v0.5.0 adds a host-managed option).
Note for GPU Operator v26.7.0+: The chart ships its own stricter copy of the resource.nvidia.com_computedomains.yaml Custom Resource Definition (CRD), which requires numNodes with no default (the DRA driver chart’s copy does not). This only matters on a fresh cluster install—Helm won’t overwrite an already-installed CRD, so existing clusters upgrading gpu-operator are unaffected. If you’re bringing up a new cluster, set numNodes explicitly in your ComputeDomain manifest, e.g. spec: {numNodes: 2, …}.
$ kubectl apply -f cd-2node-test.yaml
computedomain.resource.nvidia.com/cd-2node-test created
pod/cd-test-c01 created
pod/cd-test-c02 created
$ kubectl get computedomain,resourceclaims,pods -n cd-test
NAME STATUS NODES
cd-2node-test Ready vr-...-c01,vr-...-c02
NAME STATE AGE
cd-test-c01-imex-channel-k9j2d allocated,reserved 32s
cd-test-c02-imex-channel-l6tl2 allocated,reserved 30s
NAME READY STATUS NODE
cd-test-c01 1/1 Running vr-...-c01
cd-test-c02 1/1 Running vr-...-c02
$ kubectl logs cd-test-c01 -n cd-test # identical output on c02
crw-rw-rw- 1 root root 505, 0 channel0 # /dev/nvidia-caps-imex-channels
Going from apply to two running pods—one per compute tray, each with its IMEX channel injected and the IMEX daemons joined into a single domain—takes about 12 seconds. The same ComputeDomain path is what the training recipe’s NCCL performance gate now drives under real workload traffic: Channel injection is proven above, while the fabric bandwidth is proven in the performance phase.The entire sequence (recipe, bundle, snapshot, and the deployment and conformance validation phases) completes in about five minutes against a live cluster. The performance phase adds the runtime of the NCCL job itself.
Validation evidence and feedback loops
These results are anchored by AICR’s deployment, conformance, and performance validation across the NVL72 reference environments, paired with the successful end-to-end ComputeDomain test. Individual check details are cataloged in the validation matrix above.
Equally important is where the validated boundary sits today. The training recipe now carries a cross-node NCCL performance gate: the first performance validation on Vera Rubin silicon, exercising the same IMEX ComputeDomain path shown above under real all-reduce traffic.
Inference serving performance remains deferred for now. vLLM merged Rubin (sm_107a) support in v0.27.0, but the SM107 kernel path is gated at build time on NVIDIA CUDA >= 13.4—and vLLM’s own published images, through v0.28.0, still build against CUDA 13.0.3. So even the newest vLLM release image carries the dispatch code without the compiled kernel. ai-dynamo’s latest release (v1.4.2) pins vllm-openai v0.26.0, which predates the code entirely; its unreleased main branch pins v0.28.0, which is not built against 13.4. A VR200 inference-perf floor needs a published, CUDA >= 13.4-built runtime image verified on real hardware—past the JIT-compile crash, not just past a version check.The signed validation evidence for the Vera Rubin recipes is public: training (three phases, including the NCCL performance gate) and inference (deployment and conformance). Because the recipes are Preview and still moving, treat the published evidence as the historical record for the coordinate it was captured against rather than as an attestation of the current recipe content. A digest check against today’s recipe will report a mismatch.
The bring-up also worked in both directions. Running AICR validation on Vera Rubin surfaced an unmapped Rubin architecture value in the go-nvlib packages that crash-looped the DRA driver’s kubelet plugin. Because architecture support lands first in go-nvml; flows into go-nvlib; and from there into the GPU Operator, the Kubernetes Device Plugin, the MIG Manager, and the DRA plugins, the fix propagated consistently across the stack. It went from the first crash report to a validated build on Vera Rubin hardware in about three days. That fix (go-nvlib 0.11.0 → 0.12.0) shipped in the official DRA driver v0.5.0 release, so the Vera Rubin recipes pin the stock tag—no staging build is required.
Validation also surfaced that a host-managed IMEX daemon and DRA-managed ComputeDomains cannot share a node today, feedback that shaped the host-managed IMEX option shipped in DRA driver v0.5.0.
The Vera Rubin recipes are published in the open-source NVIDIA/aicr repository in v1.0.0, which registers the rke2 service and vr200 accelerator in the public catalog.
Get started
Get started with AICR on Vera Rubin using the Preview recipes in the AICR public catalog.
Generating the validated configuration is one command:
aicr recipe --service rke2 --accelerator vr200 --os ubuntu --intent training
For the Dynamo inference variant described above, the following command generates the recipe:
aicr recipe --service rke2 --accelerator vr200 --os ubuntu --intent inference --platform dynamo
You can follow progress, review the code, or open issues on GitHub:
NVIDIA/go-nvml
NVIDIA/go-nvlib
kubernetes-sigs/dra-driver-nvidia-gpu
NVIDIA/aicr