Meta Engineering has published details on a new multi-stage architecture designed to scale sequence learning advancements for its ads ranking platform. This architecture, detailed in an August 5, 2026 post, builds upon previous work from 2024 on sequence learning for ads recommendations, which focused on modeling the order and timing of user actions to create richer, sequence-aware representations of user interests.
The new approach introduces two architectural advancements: a multi-stage sequence model that separates heavy offline user modeling from lightweight online ranking, and a learning technique utilizing dense tokenization and target-aware attention. These innovations aim to enable LLM-style scaling laws for ads recommendations, allowing for predictable performance improvements with increased compute.
Key Points
- The architecture includes a multi-stage sequence model that decouples offline user modeling from online ranking tasks.
- A learning technique based on dense tokenization and target-aware attention is used to efficiently learn feature interactions directly from data.
- These advancements have contributed to a cumulative lift of 6% in conversions on Instagram, 3% in conversions on Facebook, and 3.5% in ad clicks on Facebook.
- The unified platform is a core component of Meta’s Generative Ads Recommendation Model (GEM).
- The offline user model processes long user histories asynchronously and produces cached embeddings.
- The online ranking model combines cached representations with real-time ad candidate signals for final ranking.
- The system demonstrates predictable scaling laws for ads recommendations, analogous to those observed in large language models.
Context
According to Meta Engineering, recommendation platforms handle billions of user interactions daily, generating temporal signals that capture user preferences. Previous approaches to sequence models often relied on hybrid configurations, where one model processed user event sequences and another handled sparse feature interactions. While effective for production demands, this hybrid method could become a bottleneck when scaling temporal sequence lengths and transformer models.
The new multi-stage model addresses this by separating the sequence model into two stages: an upstream/offline user modeling stage and a downstream/online ranking stage. The offline stage uses deep transformer models to process user-side features asynchronously, scaling to several transformer layers with sequence lengths in the thousands. These models generate embeddings that are precomputed and cached at the user level, ensuring user embeddings remain independent of any particular ad candidate. The online ranking models then use these deep representations along with fresh user signals and ad candidate information for real-time ranking, optimized for speed and latency budgets. This separation allows for increased model complexity in the offline stage without a proportional spike in serving costs for the online stage.
The dense tokenization approach integrates sparse features with sequential behavioral data into a single dense vocabulary. This allows attention mechanisms to discover interactions directly from data, unlike traditional systems that rely on manually engineered representations. Tokenized sparse features and ad candidate information are fused with user behavior sequences and processed by a memory-efficient form of multi-head attention. This enables each layer to weigh a user’s past behaviors against the specific ad being scored, progressively distilling long sequences into compact representations.
Why It Matters
This architectural shift demonstrates how large-scale recommendation systems can adopt principles from large language models to achieve predictable scaling and improved performance. For builders and researchers, it highlights a strategy for managing the tension between model complexity and serving efficiency in high-throughput, low-latency environments.
What To Do
- Note the distinction between offline user modeling and online ranking in the multi-stage architecture.
- Examine how dense tokenization and target-aware attention are used to learn feature interactions directly from data.
- Consider the implications of LLM-style scaling laws for optimizing recommendation system performance and resource allocation.
- Watch for further details on Meta’s Generative Ads Recommendation Model (GEM) and its components.