Meta Engineering (AI) has introduced MTIA 300, the initial offering in Meta's proprietary family of training and inference accelerators. This chip is specifically engineered to optimize the training of ranking and recommendation models. The design integrates built-in NIC chiplets to address the communication requirements of these models, aiming for enhanced performance compared to general-purpose GPUs.
The MTIA 300 was co-designed with HCCL, a communication library, to prioritize communication capabilities within the chip's architecture.
Key Points
- MTIA 300 is the first of Meta's in-house training and inference accelerators, optimized for ranking and recommendation models.
- The chip includes two network chiplets, each with six custom 800 Gbps RDMA NICs, providing 1.2 TB/s of total I/O bandwidth without crossing a PCIe bus.
- MTIA 300 features 16 dedicated message engines (MEs) for communication, operating independently of the 12x6 grid of processing elements (PEs) for computation.
- This design results in less than 0.5% degradation to compute throughput when running large GEMMs concurrently with collective operations, in contrast to over 20% degradation observed in traditional GPUs.
- The HCCL communication library, co-designed with MTIA 300, compiles collectives into subgraphs for autonomous execution by MEs, reducing host involvement.
- HCCL achieves up to 940 GB/s of communication bandwidth within a single rack.
- For a 150-billion-parameter production-recommendation model across 40 accelerators, MTIA 300's total communication time is 3.9 times faster than an equivalent GPU cluster.
Context
According to Meta Engineering (AI), deep learning models for personalized content, such as short videos and friend posts, have increased in complexity. This growth highlights the importance of both the compute infrastructure for training and the network connecting accelerators. Training recommendation models presents a distinct infrastructure challenge because, unlike large language models that require high floating-point throughput, recommendation models are constrained by the need for fast and efficient communication between accelerators. Their embedding tables can constitute over 99% of the model's parameters, necessitating hybrid parallelism that generates frequent collective operations across numerous accelerators.
Why It Matters
This development indicates a shift in accelerator design towards specialized hardware that addresses the unique communication bottlenecks of recommendation models. Builders and practitioners can observe how integrated networking and offloaded collective execution can improve efficiency and performance for specific AI workloads, potentially influencing future hardware and software co-design strategies.
What To Do
- Note the architectural principles of integrated networking and offloaded collective execution in MTIA 300.
- Consider how specialized hardware designs, like those in MTIA 300, address communication-intensive AI workloads.
- Watch for further details on MTIA 300's silicon design, as mentioned in the ISCA 26 conference paper.
Keep Exploring
/atlas/llama-open /atlas/gpt-family /atlas/**gemini**-family /techniques/ptcf