Sign in
← AI PulseAug 17, 2026

Wire · analysis · Single-source brief

NVIDIA Nemotron 3.5 Lightning NVFP4 Achieves 4x Throughput with QAD

NVIDIA has demonstrated that its Nemotron 3.5 Lightning model, when optimized with Quantization-Aware Distillation (QAD) and NVIDIA Model Optimizer, can achieve up to 4x higher throughput and a reduced model size of 22 GB from 66 GB.

By Illumora Editorial

Source · Aug 17, 2026, 6:12 PM · On Illumora · Aug 17, 2026, 6:18 PM

Media from the primary source — shown here so you can stay on Illumora.

Rewritten from one allowlisted primary — not independent enterprise reporting. Lanes →

Brief drafted by Illumora’s editorial model from the linked primary source. Ops desk reviews flagged pieces. How we write →

Read the source →NVIDIA Developer Blog — Developing Nemotron 3.5 Lightning NVFP4 with QAD Using NVIDIA Model Optimizer | NVIDIA Technical Blog
Save

NVIDIA has detailed a method for optimizing its Nemotron 3.5 Lightning model using Quantization-Aware Distillation (QAD) and the NVIDIA Model Optimizer. This process enables aggressive quantization to NVFP4 while maintaining accuracy close to a BF16 baseline.

The optimization results in a significant reduction in model size and an increase in throughput. The Nemotron 3.5 Lightning NVFP4 checkpoint, for example, is compressed to 22 GB from its original 66 GB full-precision size.

Key Points

  • The Nemotron 3.5 Lightning NVFP4 checkpoint achieves up to 4x higher throughput.
  • The model size is reduced from 66 GB to 22 GB through quantization.
  • QAD involves a two-stage process: post-training quantization (PTQ) followed by quantization-aware distillation.
  • QAD trains a low-precision student model against a frozen full-precision teacher using KL divergence loss and simulated quantization.
  • Experimental results show QAD consistently outperforms PTQ in median accuracy recovery and agentic/coding benchmarks.
  • NVIDIA Model Optimizer and Megatron-Bridge provide end-to-end workflows for reproducing QAD on Nemotron 3.5 Lightning.
  • The process quantizes weights to W4A16/NVFP4 while preserving accuracy near the BF16 baseline.

Context

According to NVIDIA, teams often customize models to meet specific targets for latency, speed, memory, and compute. The open NVIDIA Nemotron family of models allows developers to select an appropriately sized model for their needs. While post-training quantization (PTQ) is a common method for compression, more aggressive quantization is necessary for higher throughput and tighter memory constraints, making QAD an optimal choice.

Why It Matters

This development offers builders a method to significantly reduce the memory footprint and increase the inference speed of large language models like Nemotron 3.5 Lightning without substantial accuracy degradation. This can lead to more efficient deployment and lower operational costs for LLM-powered applications.

What To Do

  • Review the NVIDIA Developer Blog post for a detailed walkthrough of the QAD training pipeline.
  • Explore the end-to-end examples of QAD on NVIDIA Model Optimizer to understand the process.
  • Note the two-stage QAD process, including the initial PTQ phase and the subsequent distillation and evaluation.
  • Compare the performance of QAD against PTQ for aggressive quantization settings in your own models.