NVIDIA announced that its TensorRT Edge-LLM ran Qwen3.6-27B on a single NVIDIA Jetson AGX Thor Developer Kit, completing the MLPerf Inference v6.1 Edge Agentic benchmark in 24 minutes and 36 seconds. This performance is 6.4 times faster than the llama.cpp reference run, which took 2 hours and 37 minutes.
Key Points
- TensorRT Edge-LLM achieved 52.33 tokens per second on the MLPerf Inference v6.1 Edge Agentic benchmark.
- The system completed all 1,007 turns of the performance workload in 24 minutes and 36 seconds.
- This result was 6.4x faster than the llama.cpp reference run of 2 hours and 37 minutes.
- The acceleration uses NVFP4 quantization for weights and activations, FP8 KV cache, tree-based multi-token prediction, and KV cache reuse.
- KV cache and recurrent-state reuse served approximately 96% of prompt tokens from hot cache.
- Tree-based multi-token prediction delivered an additional ~40% decoding performance gain for function-calling workloads.
- The Qwen3.6-27B model used NVFP4 for weights and activations, including the language-model head, and FP8 for the KV cache.
Context
According to the NVIDIA Developer Blog, AI agents are moving from cloud data centers to edge devices, requiring models to generate tokens quickly, process long shared histories efficiently, and produce valid tool calls within limited power and memory. The MLPerf Edge Agentic benchmark measures an OpenAI-compatible model endpoint in two phases: performance and accuracy. The performance phase replays recorded software-engineering agent trajectories, involving 20 conversations and 1,007 generated turns, with input length reaching approximately 23.5K tokens.
Why It Matters
This performance improvement demonstrates how specialized optimizations can significantly enhance the efficiency of large language models on edge hardware. Builders can achieve faster agentic inference on devices like the Jetson AGX Thor, enabling more responsive and capable AI applications in environments with power and memory constraints.
What To Do
- Explore the TensorRT Edge-LLM release/0.9.1-mlpinf branch for model export settings, engine build commands, and server configuration.
- Download the calibrated Qwen3.6-27B NVFP4 checkpoint to begin deployment.
- Review the MLCommons Edge Agentic example for dataset and client configuration details.