A technical survey published on arXiv cs.CL synthesizes primary papers, official technical reports, and prior surveys to organize modern Mixture-of-Experts (MoE) systems. The survey moves beyond a chronological list of model releases to explain the architectural evolution of MoE models, which increase parameter capacity while keeping per-token computation bounded.
Key Points
- The survey organizes modern MoE systems along five coupled dimensions: expert granularity, expert topology, routing freedom, the scope of load balancing, and execution structure.
- Eight architectural milestones are described as a dependency graph, featuring six mainline developments and two orthogonal branches.
- Individual MoE systems are analyzed through four control planes: Expert Topology, Routing, Balance, and Expert Parallelism.
- These control planes specify which experts exist, which experts process each token, how aggregate load is controlled, and how selected computation maps to physical devices.
- The framework connects algorithmic choices such as Top-k routing, shared experts, fine-grained experts, and dynamic expert composition with systems concerns like token dispatch.
Context
According to the survey, the architectural evolution of MoE models cannot be fully explained by a simple chronological list of model releases. Instead, the authors propose a framework that categorizes MoE systems based on their structural and operational characteristics. This approach allows for a more nuanced understanding of how different design choices interact within these complex architectures. The survey's analysis through four control planes—Expert Topology, Routing, Balance, and Expert Parallelism—provides a structured way to examine the algorithmic and system-level decisions in MoE implementations.
Why It Matters
For builders and researchers, this survey offers a structured framework to understand and compare different Mixture-of-Experts architectures. By organizing systems along specific dimensions and control planes, it provides a clearer view of the design space, helping to inform architectural choices and identify potential areas for innovation or optimization in large language models.
What To Do
- Review the survey's proposed five coupled dimensions to categorize existing or planned MoE implementations.
- Examine the dependency graph of eight architectural milestones to understand the relationships between different MoE developments.
- Analyze specific MoE systems through the lens of the four control planes: Expert Topology, Routing, Balance, and Expert Parallelism.
- Note how algorithmic choices like Top-k routing or shared experts are connected to systems concerns such as token dispatch within the framework.