← Back to all questions
StaffAd TechML Ranking
Design a Real-Time Ad Serving System
A full Staff-level walkthrough of a real-time ad serving system that selects the best ads from millions of candidates for ~10B daily impressions in under 100ms — trading ranking quality against GPU cost through a multi-stage funnel, with second-price auctions, budget pacing, and privacy-preserving attribution. Follow it top to bottom, or jump to any step.
100% free · No login required
WHAT THIS QUESTION TESTS
·Multi-stage funnel: targeting → lightweight scoring → deep ranking → auction
·GSP vs VCG auction mechanics and clearing-price computation
·Budget pacing, frequency caps, and cross-campaign optimization
·Real-time plus batch event pipeline for impressions, clicks, and conversions
★ STAFF-LEVEL SIGNALS
★Drives funnel design from cost: GPU inference is ~50% of spend, so cut 10M candidates to ~500 before the deep model
★Quantifies auction-mechanism tradeoffs (truthfulness, revenue, complexity) and justifies a hybrid GSP/VCG choice
★Designs privacy-first attribution for a post-ATT world: aggregated measurement, server-side Conversion API, on-device signals, modeled conversions
★Plans a staged rollout: shadow scoring, 1%→100% traffic shift, auto-rollback on revenue or latency regression
0
Scope & ambiguity
“Meta’s ad system generates ~$100B+/year in revenue. The key challenge is real-time ad ranking that maximizes advertiser ROI while maintaining user experience — selecting the optimal ad from millions of candidates for each of billions of daily impressions in <100ms. I’ll focus on the auction mechanism, multi-stage ranking pipeline, pacing/budget management, and privacy-preserving attribution.”
Phases:
Phase
Timeline
Scope
Phase 1
0-3 months
Core auction engine, single-stage ML ranking, FB feed only, single region
Phase 2
3-9 months
Multi-stage ML pipeline, cross-surface (FB+IG), multi-region, targeting expansion (interest + custom audiences)
Phase 3
9-18 months
Privacy-preserving attribution, AI-generated creatives, on-device personalization
Team ownership: Auction Engine (bidding + pricing), Ranking/ML (model training + serving), Targeting (audience matching + segmentation), Ad Serving Infra (low-latency delivery + CDN), Attribution & Measurement (conversion tracking + reporting), Advertiser Platform (campaign CRUD + budgets), Integrity (ad quality + policy enforcement).
1
Requirements
Functional
- FR1: Real-time ad selection from millions of candidates per impression opportunity
- FR2: Multi-stage ranking pipeline (retrieval → lightweight scoring → deep ranking → auction)
- FR3: Auction mechanics with second-price dynamics and multi-slot allocation
- FR4: Budget pacing to spread advertiser spend evenly across campaign lifetime
- FR5: Granular targeting (demographics, interests, custom audiences, lookalikes, retargeting)
- FR6: Impression, click, and conversion tracking with real-time and batch pipelines
- FR7: Privacy compliance (ATT, GDPR, aggregated measurement, Conversion API)
Non-Functional
Requirement
Target
Ad selection latency
p50 < 50ms, p99 < 100ms
Daily impressions
~10B across FB + IG + Audience Network
Active campaigns
10M+
Peak auction throughput
200K auctions/sec
Availability
99.99%
Budget overspend
< 1% of daily budget
Feature freshness
< 1 min from user action to feature availability
2
Back-of-envelope estimation
Metric
Value
Daily impressions
~10B
Peak auctions/sec
200K
Active campaigns
10M+
Ad index size
~50GB (fits in memory per replica)
User feature store
~4TB (Redis cluster)
Daily event logs
~5TB/day (impressions + clicks + conversions)
Cost
Resource
Monthly
ML inference (GPU fleet)
~$20M
Ad index (in-memory replicas)
~$5M
Feature store (Redis cluster)
~$3M
Event processing (Kafka + Scuba)
~$4M
Storage (Hive/HDFS)
~$3M
CDN (creative delivery)
~$5M
Total
~$40M/month
Key insight: ML inference is ~50% of total cost — this drives the multi-stage funnel architecture. By filtering 10M candidates down to 500 before hitting the GPU-heavy deep ranking model, we reduce GPU spend by orders of magnitude.
3
API design
Advertiser-Facing REST
POST /v1/campaigns — Create campaign (objective, budget, schedule)
PUT /v1/campaigns/{id} — Update campaign (budget, targeting, status)
POST /v1/creatives — Upload creative (image/video + copy variants)
POST /v1/audiences — Create custom/lookalike audience
GET /v1/campaigns/{id}/insights — Reporting (impressions, clicks, conversions, ROAS)
Internal gRPC (Ad Selection)
rpc SelectAds(AdRequest) returns (AdResponse);
message AdRequest {
string user_id = 1;
string placement = 2; // FEED, STORIES, REELS, RIGHT_COLUMN
int32 num_slots = 3; // Number of ad slots to fill
UserContext context = 4; // Device, geo, session features
repeated string excluded_ad_ids = 5; // Already shown in session
}
message AdResponse {
repeated AdSlot ads = 1;
int64 latency_ms = 2;
string auction_id = 3;
}
message AdSlot {
string ad_id = 1;
string creative_url = 2;
float price_per_impression = 3; // Clearing price from auction
string tracking_token = 4; // Opaque token for attribution
}
Kafka Events
Topic: ad.events (partitioned by user_id)
Event types: IMPRESSION, CLICK, CONVERSION, AD_HIDDEN
Fields: auction_id, ad_id, campaign_id, user_id, timestamp, event_type, metadata
4
Data model
Data
Storage
Rationale
Campaign metadata
MySQL (sharded by advertiser_id)
Relational queries, transactional updates, moderate write volume
Ad index
In-memory (replicated across ad servers)
Sub-ms lookup, ~50GB fits in RAM, rebuilt every few minutes
User features (online)
Redis cluster (~4TB)
Low-latency feature serving for real-time ranking
User features (offline)
Hive/HDFS
Batch feature engineering, model training data
Event logs
Kafka → Scuba (real-time) + Hive (batch)
Real-time dashboards + offline attribution/reporting
Creatives (images/video)
Blob storage + CDN
Large binary assets, CDN for low-latency delivery
Budget counters
Redis + MySQL WAL
Redis for real-time budget checks, MySQL for durability
Privacy consent
TAO (Meta's graph store)
User consent state, fast reads, globally consistent
ML models
Blob storage + TorchServe
Large model artifacts (~100M params), GPU inference serving
5
High-level architecture
User Request (Feed Load)
|
[Feed Service]
|
[Ad Selection Service] ← dispatches to pipeline stages
|
+----+----+----+
| | |
[Candidate [Feature [Budget
Retrieval] Store] Service]
| | |
+----+----+----+
|
[ML Ranking Pipeline]
Stage 0: Targeting (~1ms, CPU) 10M → 10K-50K
Stage 1: Lightweight (~10ms, CPU) 10K → 500
Stage 2: Deep Ranking (~30ms, GPU) 500 → 50
Stage 3: Auction + Pacing (~5ms, CPU) 50 → N winners
|
[Ad Response] → Feed Service → User
|
[Event Collector] (Kafka)
| |
[Scuba [Spark/Hive
Real-time] Batch]
Failure Domains
Failure Domain
Impact
Mitigation
Ad Selection Service
Revenue loss ~$300K/hr
Fallback to cached top ads per segment, serve from stale ad index
Feature Store (Redis)
Degraded ranking accuracy
Serve with stale features (TTL-based cache), models trained to handle missing features
Budget Service
Risk overspend
Conservative fallback: reduce bid multiplier to 0.5, reconcile in batch
Event Pipeline (Kafka)
Delayed billing, no user impact
Buffer locally on ad servers, replay when pipeline recovers
6
Deep dives
WHERE STAFF IS WONAuction Mechanism: GSP vs VCG
Property
Generalized Second-Price (GSP)
Vickrey-Clarke-Groves (VCG)
How it works
Each winner pays the bid of the next-highest bidder
Each winner pays their externality (damage to others)
Truthfulness
Not truthful (bidders shade bids)
Truthful (dominant strategy to bid true value)
Revenue
Generally higher (bid shading partially offsets)
Lower but more stable
Complexity
Simple to implement and explain
Complex, harder to debug
Used by
Google Ads, Meta (historically)
Meta (for multi-slot), academic preference
Meta’s approach: GSP variant for single-slot with VCG-inspired pricing for multi-slot pages. Winner pays minimum bid needed to maintain their position.
ML Ranking Pipeline Deep Dive
For each ad impression opportunity (user loads feed):
STAGE 0 — TARGETING (~1ms, CPU):
- Boolean filters: geo match, age/gender match, placement eligibility
- Budget check: campaign active AND budget remaining > 0
- Frequency cap: user hasn't seen this ad > N times today
- Policy: ad approved, not flagged by integrity
- Input: 10M+ active ads → Output: 10K-50K eligible candidates
STAGE 1 — LIGHTWEIGHT SCORING (~10ms, CPU):
- Model: Logistic regression with sparse features
- Features: user demographics, ad category, historical CTR
- Score = bid × pCTR (expected value per impression)
- Runs on CPU — cheap enough to score 10K-50K candidates
- Input: 10K-50K → Output: top 500
STAGE 2 — DEEP RANKING (~30ms, GPU):
- Model: DLRM-style neural network (~100M parameters)
- Dense features: user embeddings, ad embeddings, cross features
- Multi-task predictions:
P(click), P(conversion), P(engagement), P(negative_feedback)
- Final score = bid × P(click) × P(conv) × conv_value
- λ × P(negative_feedback)
- Runs on GPU — expensive, only 500 candidates
- Input: 500 → Output: top 50
STAGE 3 — AUCTION + PACING (~5ms, CPU):
- Apply pacing multipliers to effective bids
- Run GSP/VCG auction for available ad slots
- Diversity rules: max 1 ad per advertiser, category spread
- Compute clearing prices (second-price)
- Input: 50 → Output: N winners (typically 1-5 per page)
Feature Categories
Category
Examples
Source
Freshness
User features
Age, gender, interests, past purchases
User profile DB
Hours
Ad features
Creative type, landing page, historical CTR
Ad index
Minutes
Context features
Time of day, device, placement, session depth
Request
Real-time
Cross features
User×Ad category affinity, user×advertiser history
Feature store
Minutes
Real-time features
Ads seen this session, clicks in last hour, current feed position
Session state
Real-time
Advertiser Targeting Deep Dive
Targeting Type
Mechanism
Latency
Match Quality
Demographic
Exact match on age, gender, location from user profile
<1ms
Exact
Interest-based
Embedding cosine similarity: user interest vector vs ad topic vector
~2ms
High (learned from engagement)
Custom audiences
Advertiser uploads hashed emails/phones, matched against Meta's user graph
Offline (pre-computed)
60-80% match rate
Lookalike audiences
FAISS approximate nearest-neighbor search on seed audience embeddings, 1-10% similarity threshold
~5ms
Moderate-high
Retargeting
Meta Pixel (web) / SDK (app) fires events, matched to user; post-ATT: Conversion API (server-side)
Minutes
High, but limited post-ATT
Retargeting frequency cap: Default 3 impressions/user/day per retargeting campaign to prevent ad fatigue.
Pacing and Budget Management
Problem: Advertiser has $1000 daily budget. If we show all their ads in the morning
(when competition is low), they run out by noon and miss the evening audience.
Solution: Budget pacing algorithm
1. For each campaign, compute target spend rate:
target_rate = daily_budget / remaining_hours_in_day
2. Every 15 minutes: compare actual spend vs target spend
- Over-pacing (spent too much): reduce bid multiplier (effectively lower priority)
- Under-pacing (spent too little): increase bid multiplier
3. Pacing types:
- Standard (even): spread budget evenly across the day
- Accelerated: spend as fast as possible (use full bid, no pacing)
- ASAP: front-load spending, slow down toward end of day
4. Real-time budget tracking:
- Every impression/click → event to Kafka → budget counter update
- Budget check before entering auction (if budget exhausted → exclude)
- Near-real-time (seconds delay) is acceptable — slight overspend OK (~1-2%)
5. Cross-campaign optimization:
- Campaign budget vs account-level daily limit
- If account approaching limit, throttle all campaigns proportionally
- CBO (Campaign Budget Optimization): ML allocates budget across ad sets
based on real-time performance signals
Privacy-Preserving Attribution
Post-iOS 14.5 world: App Tracking Transparency limits cross-app tracking.
Meta's approach:
1. Aggregated Event Measurement (AEM):
- Report conversions at aggregate level, not individual user level
- Advertiser sees: "your ad campaign drove ~500 conversions" (not which 500 users)
- Differential privacy noise added to prevent individual identification
- 72-hour delayed reporting window for aggregation
2. Conversion API (server-side):
- Advertiser sends conversion events from their server (not browser/app)
- Events matched to ad impressions using hashed identifiers (email hash, phone hash)
- Privacy: only Meta can match, advertiser sees aggregate results
- Recommended alongside Meta Pixel for maximum signal recovery
3. On-device attribution:
- iOS/Android on-device model determines which ad influenced the conversion
- Only aggregated, anonymized signal sent to Meta
- No individual-level cross-app tracking
- Uses Apple's SKAdNetwork / Privacy Sandbox Attribution Reporting API
4. Modeled conversions:
- ML model estimates unobserved conversions based on observed signals
- Fills gaps from opted-out users (~60-70% of iOS users)
- Trained on historical pre-ATT data + consented user signals
7
Multi-Team Rollout
Phase
Weeks
Deliverables
Teams
Phase 1
1-8
Core GSP auction, single-stage ML ranking (logistic regression), FB feed placement, single region (US-East), basic impression tracking
Auction Engine, Ranking/ML, Ad Serving Infra
Phase 2
9-16
Multi-stage ranking (add deep model), IG integration, interest + custom audience targeting, click/conversion event tracking, advertiser dashboard
Ranking/ML, Targeting, Attribution, Advertiser Platform
Phase 3
17-28
Budget pacing, lookalike audiences, multi-region (US-West, EU, APAC), privacy-preserving attribution (AEM + Conversion API)
Auction Engine, Targeting, Ad Serving Infra, Attribution
Migration Strategy
1. Shadow scoring: new pipeline scores alongside old, results logged but not served
2. Metric validation: compare revenue, CTR, advertiser ROAS, user negative feedback
3. Gradual traffic shift: 1% → 5% → 25% → 50% → 100%
- Each step requires 48 hours of stable metrics
4. Auto-rollback trigger: if revenue drops > 2% or p99 latency > 150ms
- Automatic revert to old pipeline within 60 seconds
- PagerDuty alert to on-call for manual investigation
8
Bottlenecks & evolution
Bottleneck
Mitigation
ML inference latency (GPU contention)
Multi-stage funnel reduces GPU candidates from 10K to 500; model distillation for smaller serving models
Ad index staleness (new campaigns delayed)
Incremental index updates every 30s via Kafka CDC, full rebuild every 5 min
Budget overspend during spikes
Pessimistic budget reservation (reserve 2x expected cost), reconcile async
Feature store failure (Redis)
Local cache with 5-min TTL on ad servers, models trained with feature dropout
Privacy regulation changes
Modular attribution layer, on-device computation ready, aggregated-by-default design
Hot advertisers (single campaign gets millions of impressions)
Shard budget counters per region, local decrement with periodic global sync
2-3 Year Evolution
Year 1: Core auction with multi-stage ML ranking, budget pacing, basic attribution, event tracking. Target: p99 < 100ms, <1% budget overspend, single-digit billion daily impressions.
Year 2: Privacy-preserving attribution at scale (AEM + Conversion API + on-device), AI-generated ad creatives (auto-generate copy/image variations via generative models), cross-platform optimization (unified FB+IG+Audience Network ad delivery), transformer-based ranking models replacing DLRM.
Year 3: On-device personalization (models run on user’s device for zero-latency ranking), privacy sandbox integration (Chrome + Android), advertiser self-serve ML tools (custom audience models, automated bidding strategies), external DSP marketplace, 30% inference cost reduction via model compression + dedicated inference ASICs.
★
Rubric — Senior vs Staff
Dimension
Senior signal
Staff signal
Ranking pipeline
Proposes an ML model to score and rank candidate ads.
Designs a cost-driven funnel (CPU filter → CPU LR → GPU DLRM → auction) and sizes each stage's fan-out and latency budget.
Auction design
Runs a second-price auction over the top ads.
Compares GSP vs VCG on truthfulness, revenue, and complexity; picks a hybrid and reasons about multi-slot clearing prices.
Budget & pacing
Stops serving a campaign once its budget is exhausted.
Builds a pacing loop with bid multipliers, near-real-time counters, account-level throttling, and bounded (~1%) overspend.
Privacy & attribution
Tracks conversions via pixel or SDK events.
Designs for ATT/GDPR: aggregated measurement with DP noise, server-side Conversion API, on-device signals, and modeled conversions.
Reliability
Adds replicas and retries to the serving path.
Defines failure domains with graceful degradation — stale ad index, feature dropout, conservative budget fallback, local event buffering.
★ MORE WALKTHROUGHS
Want more breakdowns like this?
Join free early access for upcoming RAG, LLM eval, agents, and AI infrastructure walkthroughs.