X Algorithm - The code that affects your thoughts

With this post, X open-sourced its recommendation algorithm, becoming one of the first major social media platforms to make its core feed-generation logic fully transparent.

Behind the scenes, the repository xai-org/x-algorithm replaces legacy monolithic code with a lightweight, modular microservices architecture built primarily in Rust and Python.

Architecture Overview

The repository consists of several specialized components:

• Candidate Pipeline: The foundational framework and abstraction blueprint for feed orchestration.
• Grox: The primary analysis engine that processes incoming posts, vectorizes content, and attaches metadata tags.
• Thunder: The processing engine dedicated to In-Network content (posts from accounts a user follows).
• Phoenix: The discovery engine dedicated to Out-of-Network content (posts from accounts outside a user's network).
• Home Mixer: The final orchestrator and execution layer responsible for blending, ranking, filtering, and serving the timeline.

Technical Flow & Processing Logic

When a user publishes a post on X, the payload is pushed to specific Kafka topics categorized by parameters such as timestamp, media format, and geographic region.

(Note: The initial post-creation service and Kafka push implementations are managed by internal upstream systems outside this open-source repository.)

Grox Microservice Processing

The Grox service is the first component to receive and analyze incoming posts. It performs the following operations:

• Content & Safety Analysis: Evaluates post characteristics (such as media presence and spam probability) and attaches safety metadata flags, including isNsfw, isGore, isAdult, and isSpam.
• Embedding & Media Extraction: Generates vector embeddings for the post, extracts audio from attached media, and transcribes it for contextual understanding.
• Task Routing & Tagging: Attaches processing plans and execution tags to direct downstream microservices on how the post should be ranked and handled.
• Caching Strategy: Stores processed posts and candidate data in a local cache with a 5-minute Time-To-Live (TTL) to eliminate redundant processing and prevent parallel jobs from making duplicate requests.

Data Persistence

Once processing in Grox is complete, the post metadata and analyzed candidate data are persisted into either X's Manhattan database or sent to downstream Kafka event streams, depending on the post's category.

Candidate Pipeline

This is a shared abstraction and blueprint layer for all the Rust microservices. It contains key abstractions—such as filters, hydrators, scorers, and side effects—used across the Phoenix, Thunder, and Home Mixer services.

Phoenix

Phoenix is the service responsible for fetching Out-of-Network content (posts from accounts you do not follow). It retrieves these posts using post embeddings, user metadata embeddings, and raw candidate scoring based on dot product calculations.

Detailed Workflow

Phoenix utilizes a two-tower network mechanism where user embeddings and candidate (post) embeddings share the same vector space while being stored and processed independently.

• User Embeddings: Calculated once per request based on the user's interaction history. The model evaluates up to 128 recent user interaction events within a maximum window of the last 3.3 days.
• Candidate Embeddings: Generated multimodally (incorporating text, images, and videos via a multimodal post embedder) and include contextual characteristics of the post author.
• Scoring & Selection: Calculates the dot product between the user embedding and candidate embeddings, returning the top-K results to the Home Mixer service for final ranking.

Thunder

Thunder is the dedicated service for fetching In-Network content (posts from accounts the user directly follows).

• Sub-10 ms Latency: Unlike Phoenix, Thunder skips heavy real-time machine learning inference and instead fetches in-network candidates directly from an in-memory post store.
• Recency Sorting: Candidate posts (typically around 500 candidates) are initially retrieved and sorted primarily by recency to maintain blazing-fast lookups.

Home Mixer

The Home Mixer is the ultimate gateway and execution orchestrator through which every candidate post passes before reaching the user's feed. It leverages clients for both Thunder and Phoenix to assemble the complete timeline.

Processing Pipeline

1. Candidate Tagging & Query Hydration: Marks posts as In-Network or Out-of-Network based on their source and fetches user context (engagement history, following list).

2. Data Hydration: Enriches candidate posts using specialized hydrators:

   • Core Data Hydrator: Attaches post text, repost IDs, reply counts, and other core metadata.
   • Quote Hydrator: Attaches quote posts associated with the candidate.
   • Media Hydrator: Attaches video duration and additional media metadata.

3. Filtering: Applies safety and relevance gates to remove ineligible candidates:

   • Drop Duplicate Filter
   • Age Filter
   • Self-Tweet Filter
   • Previously Seen Posts Filter
   • Muted Keyword Filter
   • Author Social Graph Filter

4. Scoring & Ranking:

   • Phoenix Scorer: Passes candidates through a transformer model to predict the probability of user actions, including P(Like), P(Reply), P(Repost), P(Quote), and P(Dwell).
   • Weighted Scorer: Combines these predicted probabilities into a single ranking score using a weighted dot product. Out-of-network candidates receive an additional scaling factor based on the oon_post_score × oon_weight multiplier.
   • Author Diversity Scorer (Guardrails): Reduces the scores of repeated posts from the same author to prevent a single account from dominating the feed.
   • (Note: Recent algorithm adjustments under Nikita Bier increased the weighting of mutual relationships, resulting in higher visibility for in-network content and stronger mutual-follower engagement.)

5. Selection & Ad Blending: Selects the top-ranked candidates and merges promoted advertisements into the timeline using dedicated ad hydrators.

6. Visibility Filtering: Removes any remaining posts that violate safety policies, visibility rules, or account restrictions.

7. Side Effects: Triggers asynchronous post-selection tasks such as analytics logging, metrics collection, and response caching before serving the final timeline to the user.