Amazon Web Services (AWS) has published a benchmark comparing the performance of small large language models (LLMs) on various GPU instances within Amazon SageMaker AI. The study focuses on G5, G6, G6e, and G7 GPU instances, evaluating their suitability for real-time LLM inference.
Key Points
- The benchmark specifically tested two 30B Mixture-of-Experts models: Qwen3-Coder-30B and NVIDIA Nemotron-3-Nano-30B.
- Testing occurred on Amazon SageMaker AI.
- The study compared throughput, latency, and cost-per-token across the selected GPU instances.
- G7 instances, which feature NVIDIA Blackwell GPUs, demonstrated measurable price-performance gains.
Context
According to the AWS Machine Learning Blog, the benchmarking aimed to provide insights into the performance characteristics of different GPU instance types for small LLM inference. The comparison included G5, G6, G6e, and G7 instances, with a focus on real-time applications.
Why It Matters
This benchmark provides data for builders to evaluate the cost-effectiveness and performance of different AWS GPU instances for deploying 30B Mixture-of-Experts models. The findings regarding G7 instances suggest potential optimizations for real-time LLM inference workloads.
What To Do
- Compare the throughput, latency, and cost-per-token data for G5, G6, G6e, and G7 instances.
- Note the specific performance gains delivered by G7 instances with NVIDIA Blackwell GPUs.
- Evaluate the suitability of these instance types for deploying Qwen3-Coder-30B and NVIDIA Nemotron-3-Nano-30B or similar 30B Mixture-of-Experts models.
- Consider the implications of these benchmarks for real-time LLM inference on Amazon SageMaker AI.