AI System Design Interview Questions
Start with high-signal AI system design rounds, then use the foundation set to sharpen the distributed-systems instincts those interviews still test.
Prioritized by AI/ML interview signal — LLM serving, RAG, agents, evals, ranking, model serving, and the foundation systems underneath.
AI system design
23 questionsDesign an LLM Inference Serving Engine
Build the GPU-resident engine behind /v1/chat/completions: continuous batching over a paged KV cache, prefill/decode scheduling, prefix-cache reuse, and speculative decoding behind per-tenant TTFT/TPOT SLOs.
Design a RAG Customer Support System
Answer support questions from a knowledge base with grounded, low-hallucination retrieval.
Design an AI Agent / Tool-Use Orchestration Platform
A multi-tenant control plane that runs LLM agents through a durable, observable plan-act-observe loop — tool registry, credential brokering, KV-aware inference, guardrails/HITL, and event-sourced replay — so long-horizon tasks complete instead of looping or leaking.
Design an LLM Evaluation Platform
Score model and prompt quality continuously with offline and online evaluation.
Design a Vector Search & Embedding Retrieval System at Scale
Turn billions of embeddings into top-k nearest neighbors in tens of milliseconds: an updatable ANN index (HNSW/IVF-PQ/DiskANN) sharded across a fleet, metadata-filtered + hybrid search, and a serving layer trading recall against latency, memory, and dollars.
Design an ML Model Serving Platform
Serve 100+ heterogeneous ML models at 5M+ predictions/sec with end-to-end p99 under 10ms, an online feature store, and GPU multi-model packing.
Design an ML Feature Store for Personalization at Scale
Serve 200M+ users feature vectors at <5ms p99 while keeping training and serving consistent across dual batch and streaming pipelines.
Design Large-Scale Distributed LLM Training Infrastructure
Train a 100B–1T+ parameter model on tens of thousands of GPUs for weeks — keeping the cluster fed, MFU high, and the run alive through a hardware failure every few hours via nD parallelism and async checkpointing.
Design an LLM Content Safety & Moderation System
Build the input/output safety layer around an LLM product: a policy taxonomy, fast classifiers, prompt-injection defense, a human-review loop, online + offline eval and red-teaming — fail-closed on the output path within a tight latency budget.
Design an LLM Pretraining Data Pipeline
Build the petabyte-scale offline batch system that turns raw web crawls into a clean, deduplicated, decontaminated, mixed, tokenized multi-trillion-token corpus a training cluster can stream without ever becoming the bottleneck.
Design a Multi-Stage Short-Video Feed Ranking System
Design the retrieval → early-rank → late-rank → re-rank funnel behind an infinite short-video feed: blend P(watch-through)/P(like)/P(share)/P(comment) into one value score under a sub-50ms budget at billions of DAU, and correct the closed-loop exposure bias created by users only ever seeing system-ranked content.
Design an Ads Ranking System with Calibrated pCTR/pCVR and Delayed Conversions
Design the ML prediction layer of an ads auction: predict pCTR and pCVR for each candidate, combine with the advertiser bid into an eCPM/expected-value score, keep both probabilities well-calibrated so billing and pacing are trustworthy, and train pCVR when conversions arrive hours-to-days late and sparse. The Staff bar is owning calibration as a first-class SLO and getting delayed-label training right, not just stacking a bigger model.
Design a Recommendation System
Serve personalized recommendations to 1B users under 200ms with a two-stage candidate-generation and ranking pipeline, ANN retrieval, and cold-start handling.
Design a Real-Time Payment Fraud & Risk Scoring System
Design the synchronous sub-100ms block/review/allow gate that scores every card transaction on a global network, where a wrong block is an instant business loss, labels arrive weeks late as chargebacks, and the counterparties adapt to beat you.
Design an Experimentation Platform with a Causal-Inference Stats Engine
Design a platform where any engineer defines treatments in code, allocates users via deterministic hashing, and a causal-inference stats engine (CUPED, sequential testing, CATE, SRM detection, FDR control) emits a trustworthy ship/no-ship decision across thousands of concurrent experiments.
Design a Fractional / Multi-Tenant GPU Sharing Platform
Partition single GPUs (MIG / MPS / memory-fraction), schedule mixed training and inference jobs with quota and borrowing, gang-schedule distributed jobs, preempt training for latency-critical inference, place jobs topology-aware over NVLink, and keep utilization high without noisy neighbors.
Design AI Copilot Infrastructure for Microsoft 365
Orchestrate RAG grounding, multi-model inference, plugins, and fail-closed safety for 300M+ M365 users.
Design a Learned ETA Prediction Service
Design the highest-QPS prediction service in a ride-hailing company: a physical routing engine emits a base ETA, and an ML model predicts the residual correction. The hard part is not the model — it is online feature freshness (live segment speeds), a sub-10ms latency budget, calibrated quantile outputs dispatch can reason about, and a deterministic fallback to the routing engine on any model timeout or failure.
Design a Visual Search System (Image-as-Query Retrieval)
Design a billion-image visual search system: a user takes a photo, you detect and crop the salient object, embed it live with a visual encoder, and retrieve visually similar or shoppable items via ANN — all under a sub-second mobile budget. The load-bearing parts are the live visual encoder, object localization/crop, and the image-to-product mapping, not a text query.
Design a Surge / Dynamic Pricing System for a Two-Sided Marketplace
Forecast short-horizon demand and supply per geo-cell, compute a surge multiplier that balances a two-sided marketplace, and serve it with guardrails (caps, smoothing, anti-oscillation) under a price-affects-demand feedback loop — while quoting consistent, idempotent prices to riders and evaluating changes with switchback experiments that survive marketplace interference.
Design an On-Device / Edge ML System
Run a model locally on a phone or vehicle chip under hard battery, memory, and thermal budgets — with OTA model distribution across heterogeneous device tiers, graceful server fallback, and privacy-preserving evaluation of a model whose inputs and outputs never leave the device.
Design a Real-Time Voice Assistant NLU Pipeline (Streaming ASR to NLU)
Design the pipeline that turns streaming ASR output into domain + intent + resolved entities and routes the turn to the right skill among thousands — inside a sub-second per-turn budget. The core difficulty is real-time speech: consuming partial hypotheses and n-best lists instead of waiting for a clean 1-best transcript, stopping ASR-error propagation through a multi-stage cascade, and holding multilingual turn latency under ~700ms end-to-end.
Design an Autonomous-Driving Data Engine (Active-Learning + Auto-Labeling Loop)
Design the closed-loop data engine behind a self-driving stack: fleet triggers that fire when the model is likely wrong, bandwidth-aware clip upload, an offline auto-labeler that uses future frames, human-in-the-loop QA on only the hard cases, and an idempotent retrain/redeploy cycle that doesn't regress. A Staff-level test of where data-centric ML meets petabyte infra.
System design foundation
22 questionsDesign a Product Search & Ranking System
Serve relevant results from a 600M-product catalog in under 200ms, balancing search relevance, purchase likelihood, and ad revenue.
Design a Streaming Analytics Platform
Ingest 1M+ events/sec, run windowed aggregations with late-event handling, and serve sub-5-second dashboards plus anomaly alerts across a multi-tenant platform.
Design a Real-Time Ad Click Aggregator
Count 50B ad clicks a day in real time with exactly-once dedup, sub-minute dashboards, and billing-grade batch reconciliation.
Design a Real-Time Ad Serving System
Select the best ad from millions of candidates for ~10B daily impressions in under 100ms, with a multi-stage ML ranking funnel, second-price auctions, and budget pacing.
Design a Real-Time Event Analytics System
Query billions of events a day with sub-second latency using in-memory columnar storage, priority sampling, and scatter-gather execution with partition pruning.
Design a Ride Dispatch & Matching System
Match riders to drivers across millions of concurrent candidates using geospatial indexing, ETA-based scoring, and batch optimization.
Design a Real-Time Ride Matching System
Match riders to the best nearby driver in under two seconds across 10K+ cities, balancing pickup ETA, driver fairness, and marketplace supply health.
Design a Distributed Task Scheduler
Place millions of containers across 100K+ machines with priority preemption, autoscaling, and multi-tenant quotas.
Design a Distributed Job Scheduler
Schedule 1M+ jobs a day across 10K machines with cron triggers, DAG dependencies, retries, and lease-based exactly-once execution.
Design a Distributed Logging & Monitoring System
Ingest 10TB/day of logs and 100B metric points across 100K+ servers with sub-minute alerting, hot/warm/cold tiering, and per-tenant chargeback.
Design a Logging and Monitoring System
Ingest 10M+ log events/sec from hundreds of microservices with full-text search, distributed tracing, metrics, and alerting that survives random instance loss.
Design a Search Autocomplete System
Serve ranked typeahead suggestions in under 100ms across ~20B keystroke requests/day with an in-memory trie, tiered caching, and real-time trending.
Design a Rate Limiter
Throttle traffic fairly across clients and protect downstream services at global scale.
Design a Feature Flag System
Roll out features safely with targeting, gradual exposure, and instant kill-switches.
Design a Zero-Trust Security Architecture for an Enterprise
Move 100K+ employees off a perimeter 'castle-and-moat' model to continuously verifying every request across identity, device, network, and data.
Design Enterprise Identity & Access Management
Issue OAuth/OIDC/SAML tokens for 700M+ identities across 50M tenants — fail-closed at five-nines, with conditional access, continuous access evaluation, and cryptographic tenant isolation.
Design a Global Identity and Access Management System
Authenticate 1B+ users and 10B+ logins a day with stateless JWTs, risk-based conditional access, and near-real-time token revocation.
Design a Real-Time Driver Location Tracking System
Ingest 1.25M GPS updates/sec from 8M+ drivers, index them with an H3 hex grid, and fan out to ride matching, live maps, and analytics under sub-second latency.
Design Real-Time GPS Tracking
Ingest 1.25M GPS updates per second from millions of drivers, serve nearby-driver lookups, and stream live location to riders within two seconds.
Design a Multi-Region Strongly Consistent Database
Keep data externally consistent across regions using TrueTime-style clocks, 2PC layered over Paxos, and zero-downtime online schema changes.
Design a Build System for a Billion-Line Monorepo
Make builds from a 1B-line monorepo fast and reproducible with remote caching, distributed execution, and impact-based test selection.
Design a Code Review System
Render diffs for thousand-file changelists in seconds, resolve OWNERS-based approvals across a monorepo, and keep comments anchored through patchsets and rebases.