Design an ML Feature Store for Personalization at Scale

“Netflix’s personalization stack runs 100+ ML models across recommendations, search ranking, artwork personalization, and content valuation. The key challenge is consistent feature computation between training and serving and low-latency online serving for real-time recommendations. I’d scope this as a platform providing a unified feature registry, dual-write pipelines (batch + streaming), and online/offline serving — so any ML team defines features once and uses them reliably across training and inference.”

Phases:

Functional

• ·FR1: Register features — define name, entity type, computation logic, freshness SLA, owner team
• ·FR2: Dual compute — execute feature logic as both batch (Spark) and streaming (Flink) from a single definition
• ·FR3: Online serving — serve feature vectors with <5ms p99 latency via EVCache
• ·FR4: Offline point-in-time — generate training datasets with features joined as-of each example's timestamp
• ·FR5: Feature catalog/discovery — searchable registry so teams find and reuse existing features
• ·FR6: Quality monitoring — track freshness, null rates, distribution drift, training-serving skew via Atlas

Non-Functional

Storage

Estimated cost: ~$450K/month (EVCache ~$200K, S3 ~$100K, Spark/Flink compute ~$150K).

Key insight: EVCache is the dominant cost and latency-critical path. Compression, TTL tuning, and tiered storage are the biggest levers.

Feature Registry REST API

Feature Serving gRPC API

Kafka topic: features.computed — emitted after each batch/streaming write with feature name, entity ID, version, timestamp, and checksum.

Data Entities

Service Boundaries

Failure Domains

Architecture (Netflix-Specific)

EVCache (Netflix’s Caching Layer)

Training-Serving Consistency

Feature Serialization Format

I’d choose Protobuf for feature vector serialization:

Protobuf wins: ~60% smaller than JSON, fast encode/decode within <5ms budget, robust schema evolution for 100+ independently-evolving model consumers.

Point-in-Time Correctness

The problem: Data leakage silently kills model quality. Features computed after the label event create signals that won’t exist at inference time — +5% offline AUC that vanishes in production A/B tests.

Implementation: Parquet partitioned by entity_type/date/ with event_timestamp column. Training data generator performs asof-join: for each example at time T, read latest partition where date <= T, filter event_timestamp <= T.

Feature Computation Pipelines

Batch Pipeline (Spark):

Streaming Pipeline (Flink):

Backfill: Dedicated lower-priority Spark job recomputes across full historical window when features are registered or computation bugs are fixed. Writes to S3 and bulk-loads into EVCache.

Failure handling:

• ·Batch fails → online store serves last-good values (TTL extended), auto-retry 3x, then page on-call
• ·Streaming fails → Flink restarts from savepoint (exactly-once), events buffered in Kafka (7-day retention) for replay

Feature Registry and Catalog

Registry (Cassandra): Versioned feature definitions tracking name, entity type, computation logic, freshness SLA, owner team, upstream sources (lineage), downstream consumers, and quality metrics.

Catalog UI: Searchable by name/entity/tag, shows distribution stats and lineage (data source → feature → models), supports access requests across teams.

Deprecation: 14-day warning to consumers → read-only mode → computation stops → S3 data retained 90 days → archived.

Feature reuse target: 60%+ — at least 60% of features in any new model from the shared catalog. This is the primary ROI justification; without reuse it’s just an expensive pipeline.

Feature Monitoring

Drift detection: Compute PSI and KL-divergence between training-time and live serving distributions. Alert at PSI > 0.25 (significant shift).

Freshness (Atlas): Track last_computed_at per feature, alert when staleness exceeds SLA. Dashboard across all 10K features sorted by staleness.

Null rate: Alert if > 5% for any feature (upstream data issue). Track coverage: % of active users with feature computed.

Training-serving skew: Periodically sample 1000 serving requests, re-compute offline, compare. Alert if divergence > 1% (consistency bug in dual materialization).

Phase 1: Foundation (Weeks 1-8)

• ·Build registry service (Cassandra) + Python/Java SDK
• ·Deploy EVCache cluster + gRPC serving API (<5ms p99)
• ·Pilot 1 team (Recommendations) — 20 features in shadow mode
• ·Success: feature values match existing pipelines within 0.1%

Phase 2: Batch Pipeline (Weeks 9-16)

• ·Spark batch pipeline + DSL compiler (definition → Spark SQL)
• ·Migrate pilot team from ad-hoc jobs to feature store
• ·Validate point-in-time join engine, launch catalog UI
• ·Success: pilot fully migrated, training data quality validated

Phase 3: Streaming Pipeline (Weeks 17-24)

• ·Flink streaming pipeline for real-time features
• ·Deploy monitoring dashboard (Atlas: freshness, drift, null rates)
• ·Onboard 2nd team (Search Ranking) — 30 features
• ·Success: streaming <30s freshness, 2 teams onboarded

Phase 4: Platform Rollout (Months 7-12)

• ·Self-service onboarding, feature sharing (catalog read-write)
• ·Migration tool (auto-converts ad-hoc Spark jobs → feature store definitions)
• ·Target: 80% models on feature store, 60%+ reuse rate

Rollback Plan

Bottlenecks

Observability (RED Metrics)

Critical Alerts

Evolution

Year 1: Foundation — core feature store with online/offline stores, batch + streaming pipelines, monitoring. Target 80% of ML models migrated from ad-hoc pipelines.

Year 2: Feature sharing catalog with cross-team discovery, automated drift detection (PSI/KL-divergence alerting), GPU-accelerated feature computation for embeddings (recomputation from 45min → 5min on GPU cluster).

Year 3: On-device feature computation for privacy-preserving personalization, AutoML integration (auto-generate candidate features from raw data), federated feature store across global regions, target 30% infrastructure cost reduction via intelligent caching and tiered storage.