X Recommendation Algorithm Architecture
With this post, X open-sourced its recommendation algorithm, becoming one of the first major social media platforms to make its core feed-generation logic fully transparent.
Behind the scenes, the repository xai-org/x-algorithm replaces legacy monolithic code with a lightweight, modular microservices architecture built primarily in Rust and Python.
Architecture Overview
The repository consists of several specialized components:
- Candidate Pipeline: The foundational framework and abstraction blueprint for feed orchestration.
- Grox: The primary analysis engine that processes incoming posts, vectorizes content, and attaches metadata tags.
- Thunder: The processing engine dedicated to In-Network content (posts from accounts a user follows).
- Phoenix: The discovery engine dedicated to Out-of-Network content (posts from accounts outside a user's network).
- Home Mixer: The final orchestrator and execution layer responsible for blending, ranking, filtering, and serving the timeline.
Technical Flow & Processing Logic
When a user publishes a post on X, the payload is pushed to specific Kafka topics categorized by parameters such as timestamp, media format, and geographic region.
(Note: The initial post-creation service and Kafka push implementations are managed by internal upstream systems outside this open-source repository.)
Grox Microservice Processing
The Grox service is the first component to receive and analyze the incoming posts. It performs the following operations:
- Content & Safety Analysis: Evaluates post characteristics (such as media presence and spam probability) and attaches safety metadata flags, including
isNsfw,isGore,isAdult, andisSpam. - Embedding & Media Extraction: Generates vector embeddings for the post, extracts audio from attached media, and transcribes it for contextual understanding.
- Task Routing & Tagging: Attaches processing plans and execution tags to direct downstream microservices on how the post should be ranked and handled.
- Caching Strategy: Stores processed posts and candidate data in a local cache with a 5-minute Time-To-Live (TTL) to eliminate redundant processing and ensure high performance.
Data Persistence
Once processing in Grox is complete, the post metadata and analyzed candidate data are persisted into either X's Manhattan database or sent to downstream Kafka event streams depending on the post's category.