NVIDIA's Developer Blog highlights that even with identical hardware, AI computing clusters can show significant differences in training throughput. The company routinely observes 8% to 12% performance gaps between partner deployments and NVIDIA's reference architectures for the same workload, model, and global batch size.
These discrepancies often stem from a combination of configuration choices across various layers, including the kernel, hypervisor, BIOS, and NVIDIA Collective Communications Library (NCCL) settings. Each misconfiguration, while potentially small, can compound to prevent deployments from meeting the 95% threshold required for NVIDIA Exemplar Cloud validation.
Key Points
- Identical NVIDIA H100, GB200 NVL72, or GB300 NVL72 systems can yield materially different training throughput.
- Performance gaps of 8% to 12% are routinely observed between partner deployments and NVIDIA reference architectures.
- Configuration issues at the kernel, hypervisor, BIOS, and NVIDIA Collective Communications Library (NCCL) levels contribute to these gaps.
- Deployments frequently miss the 95% threshold for NVIDIA Exemplar Cloud validation due to these compounded issues.
- Four recurring sources of performance loss include missing SMMU capabilities, CPU C-state and NUMA misconfiguration, insufficient NCCL queue-pair concurrency, and failure to propagate NCCL topology files.
- A GB200 NVL72 partner deployment running DeepSeek-V3 Mixture-of-Experts (MoE) FP8 pre-training experienced 12% to 14% longer iteration times in a VM compared to bare-metal reference architecture.
- Enabling CMDQV/VCMDQ in the host kernel and exposing it to the guest reduced the DeepSeek-V3 MoE iteration-time gap to within reference architecture tolerance.
Context
According to the NVIDIA Developer Blog, the material differences in training throughput across clusters built from identical hardware primarily result from compounded configuration gaps. These gaps occur at the kernel, hypervisor, BIOS, and NVIDIA Collective Communications Library (NCCL) levels. The blog details four real-world case studies that highlight recurring sources of performance loss, including issues with SMMU capabilities, CPU C-state and NUMA settings, NCCL queue-pair concurrency, and NCCL topology file propagation.
Why It Matters
Infrastructure engineers and performance architects can improve the efficiency of their AI training clusters by systematically addressing configuration details. Optimizing these settings can close performance gaps, ensuring that expensive hardware resources are utilized effectively and deployments meet validation thresholds.
What To Do
- Systematically verify SMMU and VM kernel capabilities.
- Ensure CPU power management and NUMA/process bindings are optimized.
- Tune NCCL queue-pair concurrency to match fabric scale and workload.
- Guarantee all topology and environment variables are accessible inside the intended containerized training environment.
- For Grace-based virtualized deployments, enable CMDQV/VCMDQ in the host kernel and expose it to the guest to avoid SMMU serialization.