NVIDIA has released a new capability, TensorRT multi-device inference, designed to allow a single TensorRT network to execute across multiple GPUs. This feature, supported starting with TensorRT 11.0, addresses the increasing compute and memory demands of generative AI that often exceed the capacity of a single GPU. The integration is available in NVIDIA Dynamo-Triton release 26.07, which enables the TensorRT backend's multi-device functionality.
Key Points
- TensorRT multi-device inference allows a single TensorRT network to run across multiple GPUs using NCCL-backed distributed collectives.
- This capability is fully supported from TensorRT 11.0.
- NVIDIA Dynamo-Triton release 26.07 enables the TensorRT backend's multi-device feature.
- One Triton KIND_MODEL instance can manage multiple GPUs and expose a single gRPC endpoint for distributed inference.
- The integration was demonstrated using NVIDIA Cosmos 3 Nano video generation, distributing 44,160 video tokens across up to eight GPUs.
- End-to-end generation latency for Cosmos 3 Nano dropped from 156.6 seconds on one GPU to 34.2 seconds on eight GPUs.
- The transformer RPC speedup reached 6.09x in the demonstration.
Context
According to the NVIDIA Developer Blog, the demands of generative AI frequently surpass the capabilities of a single GPU. TensorRT multi-device inference utilizes NCCL-backed distributed collectives to allow a single TensorRT network to span multiple GPUs while maintaining TensorRT inference optimizations. NVIDIA Dynamo-Triton (formerly NVIDIA Triton Inference Server) release 26.07 integrates this capability, allowing a single KIND_MODEL instance to manage multiple GPUs and present a unified gRPC endpoint for distributed inference. This eliminates the need for applications to coordinate GPU ranks directly.
The integration was showcased with NVIDIA Cosmos 3 Nano video generation, a long-sequence workload. Ulysses context parallelism distributed 44,160 video tokens across up to eight NVIDIA GPUs. The Cosmos 3 Nano model's transformer accounts for 93.4% of the single-GPU generation time, making it a primary target for acceleration. The distributed Ulysses graph is compiled into each TensorRT plan before deployment, and Dynamo-Triton activates this plan without converting a single-device engine.
Why It Matters
This integration provides a method for deploying generative AI models that require more resources than a single GPU can offer, without altering the application interface or workflow. Builders can leverage additional GPU resources to reduce request latency, package the engine as a versioned Triton model, and simplify client-side code by abstracting away rank and communicator lifecycle management. This can accelerate review-and-refine cycles for latency-sensitive generative media workflows.
What To Do
- Download NVIDIA Dynamo-Triton 26.07 from NGC to test the results in your environment.
- Review the Dynamo-Triton TensorRT backend multi-device guide for configuration details.
- Explore the TensorRT Multi-Device documentation for API and usage information.