Meta has introduced MetaRoCE, a new Remote Direct Memory Access (RDMA) transport protocol specifically engineered for AI workloads on commodity Ethernet. This development, announced on August 24, 2026, addresses the need for fast, reliable networks to move data between GPUs without wasting compute cycles, particularly for training and serving frontier AI models at scale. The company is making the MetaRoCE specification, a reference software implementation, and a compliance test available.
Key Points
- MetaRoCE is a clean-sheet RDMA transport protocol purpose-built for AI workloads on commodity Ethernet.
- It is designed to power distributed AI training at a million-GPU scale.
- The protocol moves intelligence to the endpoint, decomposing the network into fine-grained logical paths with real-time telemetry.
- MetaRoCE treats out-of-order packet arrival as normal, with each packet carrying its destination to avoid reorder buffers and head-of-line blocking.
- It operates over lossy Ethernet fabrics, not requiring features like PFC or pause frames, and uses a 256-bit selective acknowledgment bitvector for retransmission.
- The protocol combines ECN-based, sender-driven AIMD congestion control with receiver-driven fair-share rate hints.
- MetaRoCE was implemented on AMD Pensando programmable NICs for hardware validation.
- Testing on a 64-node AMD GPU cluster showed MetaRoCE delivered higher throughput and lower flow completion times than RoCEv2.
- Under 1% packet loss, MetaRoCE maintained approximately 86% throughput, and continued delivering useful bandwidth even at 10% loss rates.
Context
According to Meta, the company has been a proponent of Ethernet as the preferred fabric for AI infrastructure. They have previously demonstrated RoCE's capability in distributed AI training at scale and are now building on that work with MetaRoCE, designed for clusters of hundreds of thousands of GPUs across multiple data centers and regions. The network is critical for collective operations like all-reduce and all-to-all during training, and for low-latency communication in inference for hundreds of millions of users.
Why It Matters
This development offers builders a new networking protocol designed to enhance the efficiency and reliability of large-scale AI model training and inference. By addressing network friction and improving data transfer between GPUs, MetaRoCE aims to reduce wasted compute capacity and improve response times for distributed AI systems.
What To Do
- Review the MetaRoCE specification to understand its design principles and implementation details.
- Evaluate the reference software implementation for potential integration into existing AI infrastructure.
- Test the compliance suite to ensure compatibility and performance with current hardware setups.
- Compare the performance characteristics of MetaRoCE against RoCEv2 in environments with varying packet loss conditions.