← Back to all questions
StaffAI System DesignML InfrastructureInference
Design an ML Model Serving Platform
A full Staff-level walkthrough of a Michelangelo-style model serving platform that runs 100+ heterogeneous models at 5M+ predictions per second while holding end-to-end p99 under 10ms — budgeting every millisecond across an online feature store, GPU inference, and dynamic batching, then layering canary deployment, drift detection, and multi-model GPU packing to keep cost in check. Follow it top to bottom, or jump to any step.
100% free · No login required
WHAT THIS QUESTION TESTS
·Splits the 10ms budget into a millisecond waterfall: network, parallel feature fetch, batch wait, and GPU compute
·Selects an inference runtime for mixed TensorFlow/PyTorch/XGBoost models and wraps it in a thin orchestration layer
·Designs a dual online/offline feature store with freshness tiers and sub-2ms Redis reads
·Ships safely via validation gate, shadow traffic, progressive canary, and sub-30s rollback
★ STAFF-LEVEL SIGNALS
★Quantifies quantization tradeoffs (FP16 → INT8 → TensorRT) with concrete latency, size, and accuracy deltas
★Packs 3-5 models per GPU using CUDA MPS, QoS priority tiers, and first-fit-decreasing placement
★Kills train-serve skew with one feature DSL that compiles to both the Spark offline and Flink online jobs
★Wires PSI/KS/Wasserstein drift detection to guarded automated retraining, anchored on GPU being 62% of cost
0
Scope & ambiguity
“Uber runs 100+ ML models in production covering ride matching, ETA prediction, surge pricing, fraud detection, Uber Eats ranking, and driver incentive optimization. These models span simple logistic regressions to deep neural networks, each with different latency, throughput, and freshness requirements. The core challenge is building a unified model serving platform (Michelangelo) that can serve all of these models with p99 < 10ms while managing the full model lifecycle — from feature computation through inference to monitoring. I’ll focus on the serving architecture, feature store design, inference optimization, and production monitoring.”
Phased delivery:
Phase
Timeline
Scope
Phase 1
0-3 months
Unified serving layer for top 20 latency-critical models (ride matching, ETA, surge), online feature store, model registry with versioning
Phase 2
3-9 months
Scale to 100+ models, GPU cluster management with multi-model serving, canary deployment, A/B testing infrastructure, automated rollback
Phase 3
9-18 months
Automated retraining pipelines, drift detection, model compression toolkit, LLM serving integration, edge/on-device inference
Team ownership:
Team
Ownership
Model Serving Team
Inference engine, model loading, request routing, latency optimization
Feature Platform Team
Online/offline feature store, feature computation, feature freshness
Model Management Team
Model registry, versioning, deployment pipeline, A/B testing
ML Infrastructure Team
GPU cluster management, auto-scaling, cost optimization
ML Observability Team
Model monitoring, drift detection, alerting, automated retraining triggers
Applied ML Teams (consumers)
Individual model development, training, feature engineering
Key ambiguity I’d challenge:
“When we say p99 < 10ms, does that include feature retrieval time or just raw model inference? Because the end-to-end budget changes the architecture significantly — if features must be fetched online, we need a precomputed feature store with sub-2ms reads. If we only mean inference, we have more headroom. I’ll design for the harder case: end-to-end p99 < 10ms including feature fetch.”
1
Requirements
Functional Requirements (Phase 1)
- FR1: Serve predictions for any registered model via a unified gRPC/REST API
- FR2: Fetch precomputed features from the online feature store at inference time
- FR3: Support model versioning with instant rollback to previous versions
- FR4: Route traffic between model versions for A/B testing and canary deployments
- FR5: Provide a model registry with metadata, lineage, and deployment status
- FR6: Support both CPU and GPU inference backends depending on model complexity
Non-Functional Requirements
Requirement
Target
End-to-end prediction latency (p99)
< 10ms (feature fetch + inference + network)
Raw inference latency (p99)
< 5ms for latency-critical models
Availability
99.99% (ride matching depends on it)
Throughput
5M+ predictions/second aggregate across all models
Model deployment time
< 5 minutes from registry to serving
Feature freshness
< 1 second for real-time features (driver location), minutes for batch
Concurrent models served
100+ in production simultaneously
GPU utilization target
> 70% (GPU time is expensive)
Out of Scope (Explicitly)
- Model training infrastructure (separate Michelangelo training pipeline)
- Feature engineering tooling (data scientists write features separately)
- Offline batch prediction (different system, not latency-critical)
2
Back-of-envelope estimation
Scale Numbers
Metric
Value
Active Uber trips at peak
~1M concurrent
Prediction requests at peak
~5M/sec (each trip triggers ETA, pricing, matching, fraud models)
Unique models in production
~150
Average predictions per trip lifecycle
~50 (repeated for ETA updates, re-matching, etc.)
Feature store reads
~15M/sec (avg 3 features per prediction call)
Model sizes
1MB (logistic regression) to 500MB (deep ranking models)
Total model artifact storage
~150 models × avg 50MB × 5 versions = ~37.5GB (fits in memory)
GPU/Compute Estimation
Resource
Calculation
Result
GPU models (complex: ETA, matching, ranking)
~30 models, 2M QPS total
~200 A100 GPUs (10K inferences/sec/GPU with batching)
CPU models (simple: fraud rules, basic classifiers)
~120 models, 3M QPS total
~500 CPU instances (6K QPS/instance)
Feature store (Redis cluster)
15M reads/sec, ~50GB working set
~100 Redis shards
Model registry + metadata
Low QPS, high consistency
3-node PostgreSQL cluster
Cost Estimation
Resource
Monthly Cost
GPU cluster (200 A100s at ~$2/hr/GPU)
~$290K
CPU inference fleet (500 instances at ~$0.15/hr)
~$55K
Feature store (Redis, 100 shards)
~$80K
Network + load balancers
~$30K
Storage (model artifacts, logs)
~$15K
Total
~$470K/month
Architectural implication: GPU cost dominates at 62% of total spend. This makes GPU utilization optimization the single highest-leverage engineering investment. Multi-model GPU sharing, dynamic batching, and model quantization directly hit the bottom line. A 10% improvement in GPU utilization saves ~$29K/month.
6
Deep dives
WHERE STAFF IS WONCore Decision: Inference Runtime — TF Serving vs Triton vs ONNX Runtime vs Custom
This is the foundational technical decision. Uber’s models span multiple frameworks (TensorFlow, PyTorch, XGBoost, LightGBM), so the runtime must handle heterogeneous model formats.
Dimension
TF Serving
Triton (NVIDIA)
ONNX Runtime
Custom (Uber's approach)
Framework support
TensorFlow only
TF, PyTorch, ONNX, TensorRT, custom backends
ONNX format (converter needed)
Any (pluggable backend)
GPU optimization
Good for TF models
Best (native NVIDIA, TensorRT integration)
Good via TensorRT EP
Depends on implementation
Dynamic batching
Yes, basic
Yes, advanced (per-model config, priority)
Limited
Full control
Multi-model serving
One model per instance
Multiple models per GPU, model loading/unloading
One model per instance
Full control
Model ensemble
Not supported
Native ensemble pipelines
Not supported
Full control
Community/support
Large
Growing, NVIDIA-backed
Microsoft-backed
Internal only
Operational complexity
Low
Medium
Low
High (build + maintain)
Latency overhead
~0.5ms
~0.3ms
~0.4ms
~0.2ms (optimized for Uber)
GPU memory management
Basic
Advanced (model sharing, memory pools)
Basic
Full control
My recommendation: Triton Inference Server as the primary GPU runtime, with ONNX Runtime for CPU models, wrapped by a thin Uber-specific orchestration layer.
Rationale:
1. Triton’s multi-model GPU serving directly addresses our cost problem (GPU utilization)
2. Native dynamic batching with per-model configuration handles our heterogeneous latency requirements
3. TensorRT integration gives us the best inference performance on NVIDIA hardware
4. The orchestration wrapper handles Uber-specific concerns: feature fetching, routing, A/B testing, monitoring — things no off-the-shelf runtime covers
5. ONNX Runtime for CPU models avoids GPU overhead for simple models (fraud scoring, basic classifiers)
Why not pure custom: Uber originally built much of Michelangelo custom. The operational cost of maintaining a custom inference runtime (GPU memory management, kernel optimization, batching logic) is enormous. Triton handles the hard GPU-level problems while our wrapper handles the Uber-specific orchestration.
End-to-End Latency Budget Breakdown
Every millisecond matters when the total budget is 10ms. Here is the detailed waterfall:
Request flow for a ride-matching prediction:
[Caller Service] → [Prediction Gateway] → [Feature Fetch] → [Inference] → [Response]
Latency waterfall (p99 targets):
┌──────────────────────────────────────────────────────┐
│ 1. Network: caller → gateway │ 0.5ms │
│ 2. Request parsing + routing │ 0.3ms │
│ 3. Feature fetch (parallel) │ 2.0ms │
│ ├─ Online store (Redis): 1.2ms │ │
│ ├─ Real-time features (Flink): 1.8ms│ │
│ └─ (parallel, so max = 2.0ms) │ │
│ 4. Feature vector assembly │ 0.2ms │
│ 5. Model inference (GPU/CPU) │ 4.0ms │
│ ├─ Batch queue wait: 0.5ms │ │
│ ├─ GPU compute: 3.0ms │ │
│ └─ Result extraction: 0.5ms │ │
│ 6. Post-processing + logging │ 0.5ms │
│ 7. Network: gateway → caller │ 0.5ms │
├──────────────────────────────────────────────────────┤
│ TOTAL │ 8.0ms p99 │
│ Headroom │ 2.0ms │
└──────────────────────────────────────────────────────┘
Optimization at each stage:
Stage 1,7: Co-locate prediction service in same data center as callers
Stage 3: Precompute features, use Redis cluster with read replicas,
connection pooling (avoid TCP handshake per request)
Stage 4: Pre-allocate feature vectors, avoid memory allocation on hot path
Stage 5: TensorRT compilation (2-3x speedup), INT8 quantization,
dynamic batching (trade 0.5ms wait for 4x throughput)
Stage 6: Async logging (fire-and-forget to Kafka), don't block response
Feature Store Design
The feature store is the critical path component that most teams underestimate. Uber’s feature store must serve 15M+ reads/second with sub-2ms p99.
Architecture: Dual-store design (online + offline)
┌─────────────────────────────────────────────────────┐
│ OFFLINE STORE │
│ (Training + Batch Features) │
│ Storage: Hive/S3 (Parquet files) │
│ Latency: seconds-minutes │
│ Use: model training, batch prediction, backfill │
│ Freshness: hours (daily/hourly batch jobs) │
│ Compute: Spark jobs │
└───────────────────────┬─────────────────────────────┘
│ Materialization pipeline
│ (batch: Spark → Redis)
│ (streaming: Flink → Redis)
▼
┌─────────────────────────────────────────────────────┐
│ ONLINE STORE │
│ (Serving — Low Latency) │
│ Storage: Redis Cluster (100 shards) │
│ Latency: p99 < 1.5ms │
│ Use: real-time prediction serving │
│ Freshness: seconds (streaming) to hours (batch) │
│ Key schema: feature_group:entity_id → feature_vec │
│ Example: driver_features:driver_123 → {trip_count: │
│ 4502, rating: 4.87, online_hours_7d: 32.5, ...} │
└─────────────────────────────────────────────────────┘
Feature freshness tiers:
Tier
Freshness
Computation
Example Features
Storage
Real-time
< 1 second
Flink streaming from Kafka
Driver current location, trip state, current surge multiplier
Redis (direct write from Flink)
Near-real-time
< 5 minutes
Flink micro-batch (tumbling windows)
Driver trip count last hour, rider request frequency, area demand score
Redis (Flink sink)
Batch
< 1 hour
Spark hourly jobs
Driver lifetime rating, rider historical patterns, city-level averages
Redis (Spark materialization)
Static
Daily refresh
Spark daily jobs
City map embeddings, holiday calendars, model config
Redis (daily load)
Precomputed vs on-demand features — decision framework:
Precomputed (stored in Redis):
- Used by multiple models (amortize computation)
- Expensive to compute (aggregations, embeddings)
- High QPS (can't afford to recompute per request)
- Example: driver_lifetime_stats, rider_risk_score
On-demand (computed at request time):
- Request-specific (depends on both rider and driver in a pair)
- Cheap to compute (arithmetic, lookups)
- Low reuse across requests
- Example: distance_between_rider_driver, time_since_last_trip
Hybrid (precompute components, combine at request time):
- Rider embedding + driver embedding → cosine similarity at request time
- Area demand features + trip-specific features → combined at request time
Feature consistency between training and serving (train-serve skew prevention):
Problem: Training uses offline store (Spark), serving uses online store (Redis).
If feature computation logic differs → model accuracy degrades silently.
Solution: Feature definition DSL — single source of truth
@feature_group(name="driver_features", entity="driver_id")
class DriverFeatures:
@feature(freshness="1h")
def trip_count_7d(events: DriverEvents) -> int:
return events.filter(last_7_days).count()
@feature(freshness="realtime")
def current_lat_lng(events: LocationEvents) -> Tuple[float, float]:
return events.latest().lat, events.latest().lng
This DSL compiles to:
- Spark job (offline: reads from Hive, writes Parquet)
- Flink job (online: reads from Kafka, writes Redis)
- Both generated from SAME definition → no skew
Validation: daily job compares offline vs online feature values
for a sample of entities. Alert if divergence > 0.1%.
Model Registry Internals
The model registry is the system of record for every model artifact, its metadata, and its deployment state.
Schema (stored in PostgreSQL + S3):
Model:
model_id: UUID (PK)
name: VARCHAR (unique, e.g., "ride_matching_ranker_v3")
owner_team: VARCHAR (e.g., "marketplace-ml")
framework: ENUM (TENSORFLOW, PYTORCH, XGBOOST, LIGHTGBM, ONNX)
created_at: TIMESTAMP
ModelVersion:
version_id: UUID (PK)
model_id: UUID (FK)
version_number: INT (auto-increment per model)
artifact_uri: VARCHAR (s3://uber-models/{model_id}/{version}/)
artifact_size_mb: FLOAT
serving_format: ENUM (SAVED_MODEL, TORCHSCRIPT, ONNX, TENSORRT)
input_schema: JSONB (feature names, types, shapes)
output_schema: JSONB (prediction type, shape)
training_metrics: JSONB (auc: 0.89, rmse: 2.3, ...)
training_data: JSONB (dataset: "trips_2024q1", rows: 50M, ...)
feature_group_ids: VARCHAR[] (links to feature store definitions)
hardware_target: ENUM (CPU, GPU, GPU_TENSORRT)
status: ENUM (REGISTERED, VALIDATING, VALIDATED, DEPLOYING,
SERVING, DEPRECATED, FAILED)
created_at: TIMESTAMP
created_by: VARCHAR (user or pipeline ID)
DeploymentConfig:
deployment_id: UUID (PK)
model_id: UUID (FK)
environment: ENUM (STAGING, CANARY, PRODUCTION)
version_id: UUID (FK to ModelVersion)
traffic_percent: INT (0-100)
min_replicas: INT
max_replicas: INT
resource_request: JSONB (cpu: "2", memory: "4Gi", gpu: "0.5")
sla_latency_p99: INT (ms)
created_at: TIMESTAMP
Canary deployment protocol:
Step-by-step canary for deploying model version N+1:
1. VALIDATION GATE (automated, ~2 minutes)
- Load model artifact, run inference on golden dataset (100 examples)
- Verify: output schema matches, latency < SLA, no NaN/Inf outputs
- Compare accuracy metrics against version N (must not regress > 1%)
- If fail → status = FAILED, alert model owner, stop
2. SHADOW DEPLOYMENT (1 hour)
- Deploy version N+1 alongside version N
- Route 100% traffic to version N (source of truth)
- Fork all requests to version N+1 (shadow, responses discarded)
- Compare: prediction distributions, latency profiles, error rates
- Automated check: KL-divergence between N and N+1 outputs < 0.05
3. CANARY TRAFFIC (2-6 hours, progressive)
- Route 1% traffic to version N+1 for 1 hour
- Monitor: latency p99, error rate, business metrics (conversion, ETA accuracy)
- Gate: no regression > 0.5% on any metric
- Route 5% traffic for 1 hour
- Same monitoring and gating
- Route 25% traffic for 2 hours
- Same monitoring and gating
- If any gate fails → automatic rollback to 0% on N+1
4. FULL ROLLOUT
- Route 100% to version N+1
- Keep version N loaded for 24 hours (instant rollback capability)
- After 24 hours stable: unload version N, update status to DEPRECATED
5. INSTANT ROLLBACK (if triggered at any point)
- Switch traffic routing: 100% back to version N
- Time to rollback: < 30 seconds (routing change, no model reload needed)
- Alert on-call + model owner
- Post-mortem within 24 hours
Inference Optimization Deep Dive
Meeting p99 < 10ms with complex neural networks requires multiple optimization layers.
Quantization tradeoffs with concrete numbers:
Technique
Precision
Model Size
Inference Speedup
Accuracy Impact
Use When
FP32 (baseline)
32-bit float
100%
1x
baseline
Training, accuracy-critical offline models
FP16 (mixed precision)
16-bit float
50%
1.5-2x on A100
< 0.1% loss
Default for GPU models, nearly free perf
INT8 (post-training quantization)
8-bit integer
25%
2-4x
0.1-0.5% loss
Latency-critical models (ETA, matching)
INT8 (quantization-aware training)
8-bit integer
25%
2-4x
< 0.1% loss
When PTQ accuracy loss is unacceptable
INT4 (experimental)
4-bit integer
12.5%
4-6x
1-3% loss
Only for models with large accuracy margin
Concrete example — ETA prediction model:
FP32 baseline:
Model size: 180MB
Inference latency (single, A100): 6.2ms p99
Throughput: 1,200 inferences/sec/GPU
After FP16 conversion:
Model size: 90MB
Inference latency: 3.8ms p99 (1.6x faster)
Throughput: 2,100 inferences/sec/GPU
ETA accuracy (MAE): 2.31 min → 2.32 min (+0.4% regression, acceptable)
After INT8 quantization-aware training:
Model size: 45MB
Inference latency: 2.1ms p99 (3.0x faster than FP32)
Throughput: 4,800 inferences/sec/GPU
ETA accuracy (MAE): 2.31 min → 2.34 min (+1.3% regression, acceptable)
After TensorRT compilation (on INT8):
Model size: 38MB (fused ops, optimized graph)
Inference latency: 1.4ms p99 (4.4x faster than FP32)
Throughput: 7,200 inferences/sec/GPU
ETA accuracy (MAE): 2.34 min (no additional regression from compilation)
Result: 4.4x latency reduction, 6x throughput improvement, well within 5ms inference budget
Dynamic batching strategy with latency math:
Problem: GPU throughput scales with batch size, but batching adds wait time.
Single inference:
GPU compute time: 1.4ms (TensorRT INT8)
Throughput: ~700 inferences/sec per GPU
Batch of 8:
GPU compute time: 2.8ms (not 8x — GPU parallelism)
Per-inference time: 0.35ms
Throughput: ~2,800 inferences/sec per GPU (4x improvement)
Batch of 32:
GPU compute time: 5.5ms
Per-inference time: 0.17ms
Throughput: ~5,800 inferences/sec per GPU (8.3x improvement)
Dynamic batching configuration (per model):
max_batch_size: 32 # Never exceed (memory + latency bound)
max_batch_wait_ms: 0.5 # Wait at most 0.5ms to fill batch
preferred_batch_sizes: [8, 16, 32] # GPU-efficient sizes
priority_levels: 2 # High-priority requests get smaller batches
Latency impact:
Without batching: inference = 1.4ms, total wait = 0ms → 1.4ms
With batching: inference = 2.8ms/8 = 0.35ms, wait = 0.5ms avg → 0.85ms per request
Batching is FASTER per-request (0.85ms < 1.4ms) AND higher throughput
Caveat: at low QPS (< 100/sec), batching adds latency without throughput benefit.
→ Disable batching for models with QPS < 100/sec (use direct single inference)
Model compilation pipeline:
Artifact flow from training to optimized serving:
[Model trained in PyTorch/TF]
│
▼
[Export to ONNX format] (canonical interchange format)
│
▼
[ONNX graph optimization] (constant folding, dead code elimination,
op fusion: Conv+BN+ReLU → single kernel)
│
▼
[TensorRT compilation] (GPU models only)
│ - Layer fusion (reduce memory round-trips)
│ - Kernel auto-tuning (benchmark kernels for target GPU)
│ - Precision calibration (INT8 calibration dataset: 1000 samples)
│ - Target-specific: compiled for A100 GPU specifically
▼
[Validation]
│ - Run golden dataset (100 examples)
│ - Verify: output diff < epsilon (1e-3 for FP16, 1e-2 for INT8)
│ - Verify: latency meets SLA
│ - Verify: no numerical instability (NaN, Inf)
▼
[Upload to Model Registry]
│ - Original artifact: s3://models/{id}/original/
│ - Optimized artifact: s3://models/{id}/tensorrt-a100/
│ - Compilation metadata: {original_latency, optimized_latency, accuracy_delta}
▼
[Deploy optimized artifact to Triton]
Compilation time: 5-30 minutes (one-time cost per model version)
Pipeline trigger: automatic on new model version registration
GPU Cluster Management
With 200+ A100 GPUs at ~$290K/month, efficient resource management is critical.
Multi-model GPU serving:
Problem: Dedicating 1 GPU per model wastes resources.
- Fraud model: 10K QPS, uses 5% of GPU capacity
- ETA model: 500K QPS, uses 60% of GPU capacity
- If each gets its own GPU: fraud model wastes 95% of GPU
Solution: Multi-model GPU sharing with resource isolation
Approach: Triton's model repository with CUDA MPS (Multi-Process Service)
GPU 0 (A100, 80GB HBM):
┌────────────────────────────────────────┐
│ Model A (ETA): 8GB VRAM, 60% compute │
│ Model B (Surge): 3GB VRAM, 20% compute │
│ Model C (Fraud): 1GB VRAM, 5% compute │
│ Reserved headroom: 8GB VRAM, 15% │
│ (for model loading/swapping) │
└────────────────────────────────────────┘
GPU 1 (A100, 80GB HBM):
┌────────────────────────────────────────┐
│ Model D (Matching): 20GB, 70% compute │
│ Model E (UberEats rank): 5GB, 15% │
│ Reserved headroom: 5GB, 15% │
└────────────────────────────────────────┘
Model placement algorithm (bin-packing):
Input: list of (model, vram_requirement, compute_requirement, latency_sla)
Constraint: sum(vram) < 80% GPU memory, sum(compute) < 85% GPU compute
Objective: minimize total GPU count while meeting all SLAs
Implementation: first-fit-decreasing heuristic (rebalance weekly)
Resource isolation and QoS:
Problem: Noisy neighbor — high-throughput model starves latency-sensitive model on same GPU.
Solution: Priority tiers with compute reservation
Tier 1 (Latency-critical): ride_matching, eta_prediction, surge_pricing
- Guaranteed 70% compute share via CUDA MPS
- Preemption: can preempt Tier 2/3 inference batches
- Scaling trigger: p99 > 4ms → scale out immediately
Tier 2 (Standard): uber_eats_ranking, driver_incentive
- Best-effort compute, minimum 20% guaranteed
- Scaling trigger: p99 > 8ms → scale out
Tier 3 (Background): fraud_scoring, content_moderation
- No compute guarantee, uses idle capacity
- Can tolerate 50ms+ latency (not in critical path)
- Fallback: overflow to CPU if GPU saturated
GPU auto-scaling rules:
Scale-up: avg GPU utilization > 75% for 2 minutes → add GPU instance (< 3 min)
Scale-down: avg GPU utilization < 30% for 10 minutes → remove GPU instance
Minimum: always maintain N+1 GPUs for Tier 1 models (absorb instance failure)
Pre-scaling: predictable load patterns (rush hour 8-9am, 5-7pm) →
pre-scale 15 minutes before based on historical data
Cost optimization strategies:
Strategy
Implementation
Savings
Multi-model packing
Bin-pack 3-5 models per GPU
40-60% fewer GPUs
Quantization (FP32 → INT8)
TensorRT compilation pipeline
3-4x throughput per GPU
Spot/preemptible instances for Tier 3
Background models on spot GPUs
60-70% cost for Tier 3
Time-based scaling
Scale down 40% overnight (2am-6am)
15% overall GPU cost
Model pruning
Remove low-importance weights
20-40% smaller models, more packing
Request Routing & Load Balancing
Request flow through the routing layer:
[Caller Service]
│
▼
[Prediction Gateway] (Envoy-based, model-aware)
│
│ 1. Parse request: extract model_name, features
│ 2. Lookup routing table: model_name → deployment config
│ {
│ "ride_matching_ranker": {
│ "v5": { "traffic": 90, "endpoints": ["gpu-pool-1:8001"] },
│ "v6": { "traffic": 10, "endpoints": ["gpu-pool-2:8001"] }
│ }
│ }
│ 3. Traffic split: hash(request_id) % 100 < 10 → v6, else v5
│ 4. Route to selected version's serving endpoint
│
▼
[Model Serving Endpoint (Triton)]
A/B testing infrastructure:
A/B test configuration:
experiment:
name: "eta_model_v7_transformer"
model: "eta_prediction"
hypothesis: "Transformer architecture reduces ETA MAE by 5%"
control: { version: "v6", traffic: 90% }
treatment: { version: "v7", traffic: 10% }
stratification: city_id # ensure even distribution across cities
metrics:
primary: eta_mae (mean absolute error vs actual trip duration)
secondary: [rider_cancellation_rate, driver_acceptance_rate]
guardrail: [prediction_latency_p99, error_rate]
duration: 14 days
minimum_sample: 1M predictions per arm
significance_level: 0.05
Routing logic:
- Sticky assignment: hash(user_id + experiment_name) → consistent arm
- Multi-experiment: user can be in multiple experiments if models are independent
- Interaction detection: if two experiments modify same prediction path → mutual exclusion
Metric collection:
- Every prediction logged to Kafka: {request_id, model_version,
prediction, latency, features, timestamp}
- Outcome joiner: async Flink job joins prediction with outcome
(actual trip duration, actual surge amount, etc.)
- Statistical analysis: daily t-test on primary metric,
sequential testing for early stopping
Shadow traffic for pre-deployment validation:
Shadow mode: new model version receives forked traffic without affecting production.
Implementation:
1. Gateway duplicates request to both production and shadow endpoints
2. Production response returned to caller (normal path)
3. Shadow response logged to Kafka (never returned to caller)
4. Async comparison job:
- Distribution comparison: KS test on prediction distributions
- Latency comparison: is shadow version within SLA?
- Error comparison: does shadow version produce valid outputs?
5. Dashboard: side-by-side metrics for production vs shadow
6. Automated gate: if shadow metrics within tolerance for 4 hours →
clear for canary deployment
Shadow traffic cost: ~2x inference cost during shadow period.
Mitigation: shadow only on 10% of traffic (not 100%).
Model Monitoring & Drift Detection
Monitoring pipeline:
[Prediction Logs] → [Kafka] → [Flink Monitoring Jobs] → [Metrics Store] → [Alerts]
│
├─ Prediction distribution tracker
├─ Feature distribution tracker
├─ Latency tracker
├─ Error rate tracker
└─ Business metric correlator
Statistical drift detection methods:
Test
What It Detects
How It Works
Threshold
Alert Level
Population Stability Index (PSI)
Prediction distribution shift
Compare current hour's prediction distribution to baseline (first week)
PSI > 0.1 → warning, > 0.2 → critical
Model may be stale
Kolmogorov-Smirnov (KS) Test
Feature distribution shift
Compare current feature distribution to training distribution
p-value < 0.01 for any feature
Upstream data issue
Wasserstein Distance
Gradual drift (more sensitive than KS)
Earth mover's distance between distributions
> 2 std dev from rolling mean
Slow degradation
Error rate spike
Model failures
Count of NaN/Inf/timeout predictions
> 0.1% error rate
Immediate page
Business metric regression
Model quality degradation
Compare conversion rate, ETA accuracy, etc. to baseline
> 2% regression
Escalation to model owner
Drift detection implementation:
PSI calculation (run hourly per model):
def compute_psi(baseline_dist, current_dist, n_bins=10):
"""Population Stability Index between two distributions."""
baseline_pct = np.histogram(baseline_dist, bins=n_bins)[0] / len(baseline_dist)
current_pct = np.histogram(current_dist, bins=n_bins)[0] / len(current_dist)
# Avoid log(0) — clip to small value
baseline_pct = np.clip(baseline_pct, 0.001, None)
current_pct = np.clip(current_pct, 0.001, None)
psi = np.sum((current_pct - baseline_pct) * np.log(current_pct / baseline_pct))
return psi
# Decision logic
psi = compute_psi(baseline_predictions, current_hour_predictions)
if psi > 0.25:
trigger_alert(severity="critical",
message=f"Model {model_name} prediction drift: PSI={psi:.3f}",
action="page_oncall_and_model_owner")
trigger_shadow_retraining(model_name)
elif psi > 0.10:
trigger_alert(severity="warning",
message=f"Model {model_name} prediction drift: PSI={psi:.3f}",
action="notify_model_owner")
Automated retraining trigger:
Retraining pipeline trigger conditions (ANY of these):
1. SCHEDULED: Weekly retraining for all models (baseline freshness)
2. DRIFT DETECTED: PSI > 0.2 sustained for 6 hours → trigger immediate retraining
3. FEATURE CHANGE: Upstream feature definition changed → retraining required
4. MANUAL: Model owner requests retraining
Retraining workflow:
1. Trigger → create retraining job in Airflow
2. Fetch latest training data (last N days, configurable per model)
3. Train with same hyperparameters as current production version
4. Run evaluation on holdout set
5. Compare metrics to current production version:
- If improved → register new version → enter canary deployment
- If regressed → alert model owner, do NOT deploy
- If equivalent (within noise) → register but don't deploy (save for lineage)
6. Entire pipeline is automated — zero human intervention for standard retraining
7. Model owner reviews weekly retraining report (dashboard)
Guardrail: No model can be auto-deployed without passing the validation gate.
Maximum auto-retraining frequency: 1 per model per 24 hours (prevent thrashing).
7-8
Rollout & evolution
Rollout Strategy
Phase 1 rollout (months 1-3): Top 20 models
1. Deploy Triton cluster with 50 GPUs, onboard ETA model first (highest QPS, best-understood)
2. Feature store: migrate ETA’s features from legacy store to new Redis-backed store
3. Run shadow traffic for 2 weeks, validate latency and prediction parity
4. Canary 1% → 5% → 25% → 100% over 1 week
5. Onboard remaining 19 critical models one per week
6. Legacy Michelangelo serving runs in parallel (no decommission yet)
Phase 2 rollout (months 4-9): Remaining 100+ models
- Self-service onboarding: model teams register model, platform handles optimization + deployment
- Automated compilation pipeline (ONNX → TensorRT) for all GPU models
- GPU multi-model packing activated (target: 3-5 models per GPU)
Rollback Plan
Stage
Trigger
Rollback Action
Time to Recover
Single model regression
p99 > SLA for 5 min, or error rate > 0.1%
Auto-rollback to previous version (routing change)
< 30 seconds
GPU cluster issue
Multiple models degraded on same GPU pool
Drain affected GPU pool, route to healthy pool
< 2 minutes
Feature store outage
Feature fetch latency > 5ms or error rate > 1%
Fall back to cached features (stale but available)
< 1 minute
Platform-wide regression
New platform version degrades overall metrics
Route all traffic to legacy Michelangelo serving
< 5 minutes
Data corruption in model registry
Invalid model artifacts served
Pin all models to last-known-good versions, disable auto-deploy
< 10 minutes
Bottleneck Analysis
Component
Bottleneck
Detection
Mitigation
GPU memory
Large models cannot colocate
GPU memory utilization > 90%
Model quantization, pruning, or dedicated GPU
Feature store (Redis)
Hot-key problem (popular driver/rider IDs)
Redis SLOWLOG, key-level QPS monitoring
Read replicas, local caching in prediction gateway (TTL 100ms)
Model loading time
Large model (500MB+) takes 10+ seconds to load
Model load latency metric, cold-start timeout
Pre-warm models on standby instances, keep N+1 instances always loaded
Triton batch queue
Bursty traffic fills batch queue → timeout
Queue depth metric per model
Per-model queue limits, overflow to CPU fallback for Tier 3 models
Network bandwidth
Feature vectors for wide models (1000+ features) saturate NIC
NIC utilization monitoring
Feature vector compression, only fetch features not in local cache
Kafka logging
Prediction log volume (5M/sec × 1KB = 5GB/sec)
Kafka consumer lag
Sampling (log 10% of predictions, 100% of errors), tiered storage
Observability
Key dashboards (per model):
Metric
Granularity
Alert Threshold
Prediction latency p50/p99
Per-second, per-model
p99 > SLA for 2 min
Prediction throughput (QPS)
Per-second, per-model
Drop > 30% from baseline
Error rate
Per-minute, per-model
> 0.1%
GPU utilization
Per-second, per-GPU
< 20% (waste) or > 90% (risk)
Feature fetch latency
Per-second, per-feature-group
p99 > 2ms
Model prediction distribution
Hourly histogram
PSI > 0.2
Canary vs production metric delta
Per-hour during canary
Regression > 0.5%
Distributed tracing (end-to-end):
- Trace ID propagated from caller → gateway → feature store → Triton → response
- Each span includes: model_version, batch_size, gpu_id, feature_cache_hit_rate
- Sampled at 1% for normal traffic, 100% for errors and SLA violations
2-3 Year Evolution
Year 1: Foundation
- Unified serving platform operational for 100+ models
- Online feature store with sub-2ms reads
- GPU cluster with multi-model serving, 70%+ utilization
- Canary deployment and automated rollback
- Model monitoring with drift detection and alerting
- Operational maturity: on-call runbook, incident playbook, weekly reliability review
Year 2: Intelligence & Efficiency
- Automated retraining pipelines (zero-touch for standard models)
- Model compression toolkit (auto-quantization, distillation for new model versions)
- Advanced A/B testing with multi-armed bandit for model selection
- GPU cost optimization: 30% reduction through better packing + spot instances
- Feature store v2: support for embedding features, vector similarity lookups
- Real-time feature pipelines covering 80%+ of features (reduce batch staleness)
- Self-service model onboarding (< 1 hour from trained model to production)
Year 3: Next Generation
- LLM serving infrastructure (KV-cache management, speculative decoding, long-context)
- On-device/edge inference for latency-critical mobile predictions (e.g., driver app ETA)
- Online learning: models update weights incrementally from streaming data (no full retrain)
- Multi-modal models: single model consumes map, text, images (restaurant photos for Eats)
- Foundation model fine-tuning platform: teams fine-tune shared base models for their use case
- Cross-region model serving: serve predictions from nearest region with globally consistent feature store
- Carbon-aware scheduling: shift batch inference to regions with cleaner energy grid
★
Rubric — Senior vs Staff
Dimension
Senior signal
Staff signal
Latency budgeting
Names feature fetch and inference as the two costs under 10ms.
Builds a per-stage millisecond waterfall, parallelizes feature reads, and trades batch wait for throughput.
Inference runtime & optimization
Picks a single serving framework and batches requests.
Compares TF Serving / Triton / ONNX, justifies a Triton+ONNX split, and layers TensorRT + INT8 with measured speedups.
Feature store
Separates an online Redis store from an offline training store.
Adds freshness tiers, a precompute-vs-on-demand framework, and a DSL that eliminates train-serve skew.
Deployment safety
Supports model versioning with manual rollback.
Runs validation gate → shadow → progressive canary with automated metric gates and sub-30s routing rollback.
GPU efficiency & cost
Uses dynamic batching to raise GPU throughput.
Multi-model packs with MPS QoS tiers plus spot and time-based scaling, tying each lever back to the dominant GPU spend.
★ MORE WALKTHROUGHS
Want more breakdowns like this?
Join free early access for upcoming RAG, LLM eval, agents, and AI infrastructure walkthroughs.