← Back to all questions
StaffAI System DesignRecommendationsDistributed Systems

Design a Recommendation System

A full Staff-level walkthrough of a large-scale recommendation system that serves personalized items to 1B users under a 200ms budget — built as a two-stage pipeline that retrieves thousands of candidates with approximate nearest-neighbor search, then ranks them with a deep multi-objective model. It trades recall against latency, exploration against exploitation, and freshness against relevance, all while handling cold start and diversity. Follow it top to bottom, or jump to any step.

Level
Staff
Category
Recommendations · AI / ML
Interview time
60 min
100% free · No login required
WHAT THIS QUESTION TESTS
·Two-stage design: candidate generation then ranking
·ANN retrieval over learned user/item embeddings
·Multi-objective ranking (click + watch time + satisfaction)
·Cold start for both new users and new items
★ STAFF-LEVEL SIGNALS
★Justifies two-tower retrieval and quantifies ANN recall vs latency across HNSW/ScaNN/IVF
★Frames recommendation as exploration vs exploitation, choosing Thompson sampling over pure exploit
★Separates daily batch retraining from minute-level real-time feature freshness in the serving path
★Defends against popularity bias and filter bubbles with explicit coverage and diversity metrics
1

Requirements

Functional Requirements

  • Personalized recommendations: Show relevant items (videos, products, articles) per user
  • Multiple surfaces: Home feed, "watch next", related items, email digest
  • Real-time signals: Incorporate recent user activity (last click, last search)
  • Cold start handling: Reasonable recommendations for new users/items
  • Diversity: Avoid recommending too many similar items
  • A/B testing: Compare recommendation algorithms via controlled experiments

Non-Functional Requirements

  • Latency: < 200ms to generate recommendations
  • Scale: 1B users, 100M items, 100K recommendation requests/sec
  • Freshness: New items surfaced within hours; user preferences updated in minutes
  • Availability: 99.99% — recommendations are on the critical path
2

Back-of-envelope estimation

Metric
Estimate
DAU
500M
Recommendation requests/day
5B
QPS
~58K
Peak QPS
~150K
User feature vectors
1B users × 1KB = 1 TB
Item feature vectors
100M items × 2KB = 200 GB
User-item interaction logs/day
10B events × 100B = 1 TB/day
Model size
10-50 GB per model
3

API design

GET /api/v1/recommendations?user_id={id}&surface=home&limit=50&context={json}
Response: {
items: [{ item_id, score, reason, metadata }],
experiment_id, model_version
}
 
POST /api/v1/feedback
Body: { user_id, item_id, action: "click"|"skip"|"like"|"dislike", timestamp }
4

Data model

User Profiles: Feature Store (Redis + Bigtable)

user_features:{user_id} → {
embedding: [0.12, -0.34, ...], // 128-dim vector
recent_items: [item_1, item_2, ...],
categories: {tech: 0.8, sports: 0.3},
demographics: {country, age_bucket, language}
}

Item Features: Feature Store

item_features:{item_id} → {
embedding: [0.56, 0.23, ...],
category, tags[], popularity_score,
freshness: hours_since_publish,
quality_score, creator_id
}

Interaction Logs: Kafka → BigQuery

{ user_id, item_id, action, surface, timestamp, dwell_time, position }
5

High-level architecture

┌──────────┐ ┌──────────────┐ ┌──────────────────┐
│ Client │────▶│ API Gateway │────▶│ Recommendation │
│ │ └──────────────┘ │ Service │
└──────────┘ └────────┬─────────┘
│
┌─────────────────┼────────────────┐
│ │ │
┌─────┴──────┐ ┌───────┴──────┐ ┌──────┴───────┐
│ Candidate │ │ Ranking │ │ Re-ranking │
│ Generation │ │ Service │ │ (diversity, │
│ (retrieve │ │ (score │ │ business │
│ 1000s) │ │ top 100) │ │ rules) │
└─────┬──────┘ └───────┬──────┘ └──────────────┘
│ │
┌────────┼──────┐ ┌─────┴──────┐
│ │ │ │ ML Model │
ANN Index CF Popular │ Serving │
(Embeddings) │ │ (TF Serving)│
Feature Store └────────────┘
(Redis/Bigtable)
 
┌──────────────────────────────────────────────┐
│ Offline Training Pipeline │
│ Interaction Logs → Feature Eng → Training │
│ → Model Registry → A/B Experiment Config │
└──────────────────────────────────────────────┘

Two-Stage Architecture

Stage 1 — Candidate Generation (retrieve ~1000 from millions):

  • Collaborative filtering: Users who interacted with similar items
  • Content-based: Items similar to user's history (embedding similarity)
  • Popularity: Trending items in user's region
  • Social: Items friends interacted with
  • Uses ANN (Approximate Nearest Neighbor) index for fast retrieval

Stage 2 — Ranking (score and order top 100):

  • Deep neural network scores each candidate
  • Features: user features, item features, context (time, device), cross-features
  • Output: predicted engagement probability (click, watch time, like)

Stage 3 — Re-ranking (apply business rules):

  • Diversity: limit items per category/creator
  • Freshness boost: mix in new items
  • Business rules: promoted content, content policy filtering
6

Deep dives

WHERE STAFF IS WON

Deep Dive 1: Candidate Generation with ANN

Problem: Find ~1000 relevant items from 100M items in < 50ms.

Approach: Learn user and item embeddings; find nearest items to user embedding.

Embedding models:

  • Matrix Factorization: Decompose user-item interaction matrix. Simple, effective.
  • Two-Tower Model: Separate neural networks for user and item, trained with contrastive loss. More expressive.

ANN search algorithms:

Algorithm
Build Time
Query Time
Memory
Recall
Brute force
O(1)
O(n)
O(n)
100%
LSH
O(n)
O(1)
O(n)
~90%
HNSW (Hierarchical NSW)
O(n log n)
O(log n)
O(n)
~98%
IVF (Inverted File)
O(n)
O(√n)
O(n)
~95%
ScaNN (Google)
O(n)
O(1)
O(n)
~99%

Our choice: HNSW or ScaNN — best recall/speed tradeoff. Index updated hourly with new items.

Deep Dive 2: Ranking Model

Architecture: Deep & Cross Network (DCN) or Wide & Deep

Features:

  • User: embedding, demographics, activity recency, preferences
  • Item: embedding, category, popularity, freshness, quality score
  • Context: time of day, device type, surface (home vs search)
  • Cross-features: user_category × item_category interaction

Training:

  • Objective: multi-task (predict click + predict watch time + predict like)
  • Data: billions of user-item interaction logs
  • Updated: retrained daily; online learning for real-time signal integration

Serving: TF Serving / TorchServe with GPU inference

  • Batch scoring: score 1000 candidates in single forward pass
  • Latency: < 50ms for batch of 1000

Deep Dive 3: Cold Start Problem

New users (no interaction history):

  • Use demographic-based recommendations (popular in user's country/age group)
  • Content-based on sign-up interests
  • Rapidly incorporate first few interactions (online learning)

New items (no interaction data):

  • Content-based features only (title, description, category)
  • Exploration: inject new items into some users' feeds with lower ranking weight
  • Multi-armed bandit: allocate exploration budget to learn item quality quickly
7

Bottlenecks & tradeoffs

Key Tradeoffs

Decision
Option A
Option B
Choice
Architecture
Single model (simple)
Two-stage (candidate gen + ranking)
Two-stage (scalable)
Candidate gen
Collaborative filtering only
Multi-source (CF + content + popular)
Multi-source (better coverage)
Model update
Batch (daily)
Online (real-time)
Batch + real-time feature updates
Ranking objective
Click prediction
Multi-objective (click + watch time + satisfaction)
Multi-objective
Exploration
None (exploit only)
Epsilon-greedy / Thompson sampling
Thompson sampling (principled exploration)

Monitoring

  • Recommendation quality: Click-through rate, watch time, user satisfaction surveys
  • Serving latency p99: Alert if > 500ms
  • Model staleness: Age of current model; alert if > 48 hours
  • Coverage: % of items ever recommended; low coverage = popularity bias
  • A/B test metrics: Statistical significance monitoring, guardrail metrics
★

Rubric — Senior vs Staff

Dimension
Senior signal
Staff signal
Architecture
Proposes one ranking model scoring the catalog.
Splits retrieval from ranking and justifies a two-stage pipeline that scales to 100M items.
Candidate generation
Relies on collaborative filtering alone.
Blends CF, content, popularity, and social sources; picks an ANN index by recall/latency budget.
Ranking model
Predicts click probability with a single objective.
Uses a multi-objective DCN/Wide&Deep with cross-features and batched GPU scoring under 50ms.
Cold start & exploration
Falls back to popular items for new users.
Adds bandit-driven exploration budget and online learning to absorb first interactions fast.
Quality & guardrails
Tracks click-through rate.
Monitors coverage, diversity, model staleness, and A/B guardrails to catch popularity bias.

Related questions

★ MORE WALKTHROUGHS

Want more breakdowns like this?

Join free early access for upcoming RAG, LLM eval, agents, and AI infrastructure walkthroughs.

Join Free Early Access →