Alibaba has released the model weights for Qwen3.8-Flash-Next, a 176B parameter multimodal Mixture-of-Experts (MoE) model. This release serves as a preview of the upcoming Qwen4 architecture, allowing developers to experiment with and evaluate the model. Qwen3.8-Flash-Next features a 262,144-token context window, which is extensible to 1M tokens using YaRN.
Key Points
- Qwen3.8-Flash-Next is a multimodal Mixture-of-Experts (MoE) model with 176B total parameters, activating 6B parameters per token.
- It introduces a hybrid Gated DeltaNet (GDN) and Qwen Sparse Attention (QSA) architecture to address long-context inference bottlenecks.
- GDN compresses historical context into a fixed-size recurrent state, while QSA aggregates sequence tokens into micro-blocks for importance estimation.
- Benchmarks show QSA provides up to 7.6x prefill speedup and 4.9x decoding speedup over full attention.
- Qwen3.8-Flash-Next achieved 8.6x the prefill throughput of Qwen3.7-Plus at 1M-token contexts in a cache-heavy online serving test.
- Running on NVIDIA GB300 NVL72, the model achieves over 16K tokens/sec per GPU.
- NVIDIA provides Day 0 functional support through SGLang, vLLM, and NVIDIA TensorRT LLM, and post-training recipes from NVIDIA NeMo AutoModel and NVIDIA NeMo RL.
Context
According to the NVIDIA Developer Blog, Qwen3.8-Flash-Next is designed for high-volume, context-intensive applications such as agentic coding, document processing, and tool-driven workflows. The model's hybrid GDN and QSA architecture specifically addresses the bottlenecks of attention compute and KV cache memory that arise as context grows. Three out of every four layers use GDN to continuously compress historical context, while the remaining layer uses QSA for precise retrieval across the full context.
Why It Matters
The Qwen3.8-Flash-Next model offers developers a preview of the Qwen4 architecture, demonstrating advancements in handling long contexts and improving inference efficiency. Its performance on NVIDIA GB300 NVL72 and support for various NVIDIA tools indicate a path for scaling agentic coding and other context-intensive applications from local hardware to rack-scale deployments.
What To Do
- Download the model weights from Hugging Face or ModelScope to begin experimentation.
- Explore NVIDIA NeMo AutoModel for fine-tuning the model for domain-specific use cases.
- Investigate NVIDIA NeMo RL recipes for reinforcement learning applications.
- Test inference performance using SGLang, vLLM, or TokenSpeed on NVIDIA-accelerated platforms.
- Note the model's performance on NVIDIA GB300 NVL72 for potential production serving.