← Back to all questions
SeniorStaffDistributed SystemsScalability
Design a Rate Limiter
Throttle traffic fairly across clients and protect downstream services at global scale.
100% free · No login required
WHAT THIS QUESTION TESTS
·Algorithm choice: token bucket vs sliding window
·Distributed counters and clock skew
·Fairness, isolation, and cost of coordination
★ STAFF-LEVEL SIGNALS
★Quantifies the accuracy vs cost tradeoff of centralized counters vs local buckets with async sync.
★Designs for clock skew and boundary bursts (sliding-window-counter) instead of naive fixed windows.
★Defines a fail-open vs fail-closed policy per endpoint when the counter store is unavailable.
★Plans multi-region: local enforcement with eventual reconciliation, not one synchronous global counter.
1
The prompt
Design a rate limiter that throttles requests across many clients and protects downstream services at global scale. It must enforce per-client and per-endpoint quotas, stay fast on the hot path, behave correctly across many distributed nodes, and degrade safely when its own storage is slow or unavailable.
2
Senior-level outline
- Clarify limits: dimensions (user / IP / key / endpoint), window sizes, and whether limits are hard or soft.
- Pick an algorithm and justify it: token bucket for smooth bursts, sliding window for accuracy.
- Centralized counters in an in-memory store (Redis) with atomic increment + TTL per window.
- Return standard signals: HTTP 429 with Retry-After and rate-limit headers.
- Estimate load: peak QPS, key cardinality, memory per counter, and added round-trips.
- Enforce at the edge / gateway so rejected traffic never reaches core services.
3
Common mistakes
- Jumping to an algorithm before clarifying what is being limited and how strictly.
- Using a naive fixed window and ignoring the 2× burst at window edges.
- Assuming a single global counter scales — ignoring latency and coordination cost.
- No answer for race conditions on concurrent increments.
- Forgetting the failure mode: what the limiter does when Redis is down.
◇
AI engineering connection
A rate limiter is the backbone of running LLM products. The same primitives — token buckets, per-tenant quotas, and fail-safe degradation — are how you cap inference spend, stop abuse and prompt-injection floods, enforce per-customer usage tiers, and shed load gracefully when a model provider is slow. AI Engineer and AI Infrastructure interviews reuse this exact reasoning, just with tokens-per-minute and GPU cost as the scarce resource instead of plain requests.
★
Rubric — Senior vs Staff
Dimension
Senior signal
Staff signal
Requirements
Identifies limit dimensions and window
Separates hard vs soft limits, negotiates per-tier SLAs
Algorithm
Names token bucket / sliding window
Justifies choice with an accuracy vs cost tradeoff
Scalability
Centralized counter with TTL
Sharding, hot-key handling, multi-region strategy
Consistency
Uses atomic increments
Handles clock skew, boundary bursts, races deliberately
Reliability
Mentions store failure
Explicit fail-open / fail-closed policy and blast radius
Tradeoffs
Notes one tradeoff
Quantifies coordination cost vs accuracy
Ownership
Mentions 429 metrics
Owns hot-key alerts, fairness, tenant isolation
★ MORE WALKTHROUGHS
Want more breakdowns like this?
Join free early access for upcoming RAG, LLM eval, agents, and AI infrastructure walkthroughs.