← Back to all questions
SeniorStaffReliability

Design a Notification Platform

A full Staff-level walkthrough of a system that delivers hundreds of millions of multi-channel messages a day — with no duplicates and strict per-channel SLAs. Follow it top to bottom, or jump to any step.

Level
Senior / Staff
Category
Reliability · Messaging
Interview time
45 minutes
100% free · No login required
WHAT THIS QUESTION TESTS
·Idempotent delivery & deduplication
·Backpressure, retries & dead-letter handling
·Per-channel SLOs & graceful degradation
·Operational ownership in production
★ STAFF-LEVEL SIGNALS
★Quantifies tradeoffs & chooses deliberately
★Failover providers & circuit breakers
★Graceful degradation & priority shedding
★Owns SLOs, rollout & on-call
0

Scope & ambiguity

Before drawing a single box, separate transactional (password resets, receipts) from marketing (campaigns, digests) — they have different latency, priority, and failure tolerances. Pin delivery semantics: target at-least-once delivery with deduplication, not true exactly-once. Define what “never duplicate” actually means: idempotent within a bounded dedup window.

★ STAFF SIGNAL
Naming the exactly-once tradeoff out loud — and deliberately choosing at-least-once + dedup — is the difference between a Senior and a Staff opening.
⚠ COMMON TRAP
Jumping straight to boxes-and-arrows before clarifying delivery guarantees. You’ll design the wrong system.
1

Requirements

FUNCTIONAL
·Email, SMS, push — one API
·Templates & scheduling
·User preferences & opt-out
NON-FUNCTIONAL
·~300M notifications / day
·No duplicates; per-channel SLOs
·Transactional push p99 < 2s
★ STAFF SIGNAL
Preferences and opt-out aren’t a “nice to have” — they’re a compliance requirement (CAN-SPAM / TCPA). Surfacing them here signals product and legal awareness.
2

Back-of-envelope estimation

Size the system out loud so every later decision has a number behind it.

capacity.txt
avg = 300M/day ÷ 86,400s ≈ 3.5K msg/s
peak = avg × 5 (campaign bursts) ≈ 17K msg/s
log = 300M × 1KB × 30d ≈ 9 TB // delivery history
★ STAFF SIGNAL
The 5× peak factor is the tell. A Staff candidate sizes the queue and worker pool for the burst, not the average — campaigns are exactly when the system breaks.
3

API design

A single ingest endpoint, with the idempotency_key carried at the boundary so retries from callers collapse safely.

POST /v1/notifications
{
"idempotency_key": "ntf_9f2a4c…",
"channel": "push" | "sms" | "email",
"template_id": "order_shipped_v3",
"recipient": "usr_8821",
"priority": "transactional" // sheds last
} → 202 Accepted, { notification_id }
★ STAFF SIGNAL
Returning 202 Accepted with an id (not a synchronous send) is what makes the platform async, queueable, and resilient under burst.
4

Data model

TABLE
PARTITION KEY
STORE
notification
idempotency_key
Cassandra
delivery_attempt
(ntf_id, provider)
Cassandra
template
template_id
Postgres
user_preference
user_id
Postgres + cache
★ STAFF SIGNAL
Using idempotency_key as the partition key makes dedup a free side effect of the write path — a retry storm collapses to a single row, no separate dedup service required.
5

High-level architecture

API Gateway→Idempotency / dedup→Kafka (per-priority)→Channel workers→Provider adapters
supporting:SchedulerPreference serviceStatus storeDead-letter queue
◇ AI ENGINEERING CONNECTION
Swap “provider adapters” for “LLM provider adapters” and this is an inference gateway: queue by priority, route to model providers, track per-request status, retry on failure.
6

Deep dives

WHERE STAFF IS WON

Spend the most time here. Interviewers calibrate level on the depth of these.

A · Idempotency & exactly-once cost

At-least-once + dedup-on-write gets you effectively-once at a fraction of the coordination cost. True exactly-once needs distributed transactions across provider + store — defend why that price isn’t worth it here.

B · Provider failover & circuit breakers

Each channel runs primary + fallback providers behind a circuit breaker. When a provider’s deliverability or latency degrades, trip the breaker and reroute — don’t pile retries onto a failing dependency.

C · Backpressure & priority shedding

Separate queues per priority. Under overload, shed marketing first to protect transactional SLAs — graceful degradation, not collapse.

★ STAFF SIGNAL
“Shed marketing before transactional” is a textbook Staff move — it shows you optimize for business-critical SLAs under failure, not uniform fairness.
⚠ COMMON TRAP
Having no story for third-party provider failure. “We’d retry” isn’t enough — retries without a breaker amplify an outage.
7

Rollout strategy

Migrate off the existing system with zero downtime: shadow-write to the new platform first, compare delivery outcomes, then cut over one notification type at a time behind a feature flag — with an instant kill-switch back to the old path.

★ STAFF SIGNAL
Proposing a shadow + per-type cutover with a kill-switch — rather than a big-bang migration — is pure Staff-level operational ownership.
8

Bottlenecks, observability & evolution

Bottlenecks
Hot partitions on popular templates; provider rate limits. Mitigate with key salting & per-provider token buckets.
Observability
Per-channel SLO dashboards, deliverability alerts, and a delivery funnel: queued → sent → delivered → failed.
Evolution
Multi-region active-active, new channels (WhatsApp, in-app), and self-serve onboarding for product teams.
◇ AI ENGINEERING CONNECTION
The delivery funnel maps directly onto an LLM eval funnel — requested → generated → grounded → accepted — and the same SLO/alerting discipline is what AI Infra interviews probe for.
★

Rubric — Senior vs Staff

Dimension
Senior signal
Staff signal
Requirements
Captures channels, volume, latency
Separates transactional vs marketing, negotiates SLAs
Architecture
Clean queue + worker design
Justifies each component against alternatives
Scalability
Estimates QPS and scales workers
Plans for hot partitions, fan-out storms, multi-region
Reliability
Retries + dead-letter queue
Idempotency, failover providers, graceful degradation
Tradeoffs
Names one or two tradeoffs
Quantifies tradeoffs and chooses deliberately
Communication
Clear, structured walkthrough
Leads the discussion, checks in, adjusts depth
Staff-level ownership
Mentions monitoring
Owns SLOs, rollout, on-call, cross-team enablement

Related questions

★ MORE WALKTHROUGHS

Want more breakdowns like this?

Join free early access for upcoming RAG, LLM eval, agents, and AI infrastructure walkthroughs.

Join Free Early Access →