← Back to all questions
StaffStream ProcessingData Infrastructure

Design a Streaming Analytics Platform

A full Staff-level walkthrough of a streaming analytics platform that ingests 1M+ events/sec, runs windowed aggregations over continuous streams, and serves sub-5-second dashboards with anomaly alerting — trading processing latency against accuracy as it handles late events, exactly-once billing streams, and hot/warm/cold storage tiers. Follow it top to bottom, or jump to any step.

Level
Staff
Category
Stream Processing · Data Infrastructure
Interview time
60 min
100% free · No login required
WHAT THIS QUESTION TESTS
·Windowed aggregations: tumbling, hopping, sliding, session
·Watermarks and a correction pipeline for late, out-of-order IoT events
·At-least-once + idempotent sinks broadly; exactly-once for billing
·Hot/warm/cold tiering with retention from Event Hubs to Cosmos DB to Data Lake
★ STAFF-LEVEL SIGNALS
★Splits managed SQL stream processing for simple jobs from Flink for complex stateful ones
★Quantifies partition-count tradeoffs (32 vs 128 vs 256) against rebalance time and hot spots
★Change-feed-driven alert evaluation — decoupled yet reactive, avoiding polling overhead
★Controls Flink state growth with RocksDB incremental checkpoints and TTL on long windows
1

Requirements

Functional Requirements

  • Event ingestion: Accept high-throughput events from IoT devices, mobile apps, web applications, and backend services
  • Stream processing: Windowed aggregations (tumbling, hopping, sliding, session) on continuous event streams
  • Real-time dashboards: Sub-5-second latency dashboards with drill-down from overview to individual events
  • Anomaly alerting: Threshold-based and statistical anomaly detection with PagerDuty/Teams/email notification
  • Data lake integration: Archive all raw events to Azure Data Lake for batch analytics and ML training
  • Materialized views: Pre-computed aggregations queryable via REST API for downstream services

Non-Functional Requirements

  • End-to-end latency: < 5 seconds from event emission to dashboard update
  • Ingestion throughput: 1M+ events/sec sustained, 5M+ events/sec burst
  • Processing guarantee: At-least-once for most pipelines; exactly-once for billing/financial streams
  • Availability: 99.9% for processing; 99.99% for ingestion
  • Retention: 7 days hot (Event Hubs), 90 days warm (Cosmos DB), 7 years cold (Data Lake)

Constraints

  • Event ordering is per-partition only — no global ordering guarantee
  • Late events from IoT devices with intermittent connectivity must be handled gracefully
  • Multi-tenant: multiple teams share the platform with resource isolation
2

Back-of-envelope estimation

Metric
Estimate
Sustained event rate
1M events/sec
Average event size
500 bytes (JSON/Avro)
Ingestion bandwidth
1M × 500B = 500 MB/s
Daily raw volume
500 MB/s × 86,400 = ~43 TB/day
Event Hubs TUs needed
500 MB/s / 1 MB/s per TU = 500 TUs
Peak burst (5x)
5M events/sec = 2.5 GB/s
7-year cold storage (raw)
43 TB/day × 365 × 7 = ~110 PB
7-year cold storage (Parquet 5x compression)
~22 PB
Cosmos DB (90-day materialized views)
~50 TB
Dashboard queries/sec
~10K (across all teams)
Unique event sources
~10M (devices + services)
3

API design

# Ingest events (batch)
POST /api/v1/events
Body: { events: [{ event_id, event_type, event_time, source_id, payload: {} }] }
Response: { accepted: 1000, failed: 0 }
 
# Bulk ingest (Avro/JSON, up to 1MB)
POST /api/v1/events/batch
Body: Binary Avro batch
Response: { accepted, failed, errors }
 
# Query materialized aggregations
GET /api/v1/metrics?metric={name}&source={id}&from={ts}&to={ts}&window={1m|5m|1h}
Response: { data_points: [{ window_end, value, count }], next_cursor }
 
# Create alert rule
POST /api/v1/alerts
Body: { name, metric, condition, threshold, window_minutes, channels: ["pagerduty", "teams"] }
Response: { alert_id }
 
# Alert trigger history
GET /api/v1/alerts/{alert_id}/history?from={ts}&to={ts}
Response: { triggers: [{ triggered_at, value, resolved_at }] }
 
# Deploy stream processing pipeline
POST /api/v1/pipelines
Body: { name, query_sql, input_topic, output_sinks: ["cosmosdb", "powerbi"] }
Response: { pipeline_id, status }
 
# Ad-hoc query
POST /api/v1/queries
Body: { sql, time_range: { from, to } }
Response: { results: [...], execution_time_ms }
4

Data model

Event Schema: Avro (in Event Hubs with Schema Registry)

{
"type": "record",
"name": "StreamEvent",
"fields": [
{ "name": "event_id", "type": "string" },
{ "name": "event_type", "type": "string" },
{ "name": "event_time", "type": "long" },
{ "name": "source_id", "type": "string" },
{ "name": "payload", "type": { "type": "map", "values": "string" } }
]
}

Materialized Views: Cosmos DB

{
"id": "sensor_123_2024-01-15T10:05:00Z",
"source_id": "sensor_123",
"window_end": "2024-01-15T10:05:00Z",
"window_size_minutes": 5,
"avg_temperature": 78.3,
"max_temperature": 82.1,
"event_count": 150,
"partition_key": "sensor_123"
}

Cold Storage: Azure Data Lake Gen2 (Parquet)

adl://analytics-lake/raw/{year}/{month}/{day}/{hour}/{partition_id}.parquet
adl://analytics-lake/aggregated/{metric}/{year}/{month}/{day}/results.parquet

Alert Rules: PostgreSQL

CREATE TABLE alert_rules (
rule_id SERIAL PRIMARY KEY,
name VARCHAR(200),
metric VARCHAR(100),
condition VARCHAR(20), -- 'gt', 'lt', 'anomaly'
threshold FLOAT,
window_minutes INT,
channels TEXT[],
cooldown_minutes INT DEFAULT 15,
enabled BOOLEAN DEFAULT TRUE
);

Technology Choices

  • Event Hubs: Kafka-compatible, auto-inflate, native Azure integration
  • Cosmos DB: Global distribution, single-digit-ms reads, partition key = source_id
  • Data Lake Gen2: Hierarchical namespace, optimized for Parquet/ORC, cost-effective cold storage
  • PostgreSQL: Alert rule configuration (low volume, relational queries)
5

High-level architecture

┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ IoT Hub │ │ App Service │ │ Custom SDK │
│ (devices) │ │ (web/mobile) │ │ (services) │
└──────┬───────┘ └──────┬───────┘ └──────┬───────┘
│ │ │
└─────────────────┼─────────────────┘
▼
┌──────────────────────┐
│ Azure Event Hubs │
│ (partitioned topics) │
└──────────┬───────────┘
│
┌─────────────┼──────────────┐
▼ ▼ ▼
┌────────────┐ ┌──────────┐ ┌───────────┐
│ Azure │ │ Apache │ │ Capture │
│ Stream │ │ Flink │ │ (archiver)│
│ Analytics │ │ (complex)│ │ │
└──────┬─────┘ └────┬─────┘ └─────┬─────┘
│ │ │
┌──────┴─────┐ ┌────┴─────┐ ┌─────┴──────┐
│ Power BI │ │Cosmos DB │ │ Data Lake │
│ (dashboard)│ │(mat.view)│ │ (cold) │
└────────────┘ └──────────┘ └────────────┘
│ │
└──────┬─────┘
▼
┌──────────────┐
│ Alert Service │──▶ PagerDuty / Teams / Email
└──────────────┘

Data Flow: Real-Time Metric Dashboard

1. IoT device emits temperature reading → IoT Hub

2. IoT Hub routes event to Event Hubs topic (partitioned by source_id)

3. Azure Stream Analytics job applies 5-minute tumbling window aggregation

4. Aggregated result pushed to Power BI streaming dataset

5. Dashboard tile updates within 5 seconds of window close

6. If threshold breached (avg temp > 80°), alert pushed to Azure Monitor → PagerDuty

Data Flow: Late Event Correction

1. IoT device reconnects after connectivity outage, sends buffered events

2. Events arrive at Event Hubs with event_time before current watermark

3. Flink identifies events as late (event_time < watermark)

4. Late events routed to correction pipeline

5. Cosmos DB materialized view updated with corrected aggregation

6. Correction event published for downstream consumers to reconcile

6

Deep dives

WHERE STAFF IS WON

Deep Dive 1: Event Hubs Partitioning and Ingestion

The challenge: Ingest 1M+ events/sec with ordering guarantees and replay capability.

Partition key strategy:

  • IoT telemetry: partition by source_id (events from same device stay ordered)
  • Multi-tenant: partition by tenant_id (isolation between teams)
  • High-cardinality sources: hash-based partition assignment across 128+ partitions

Consumer group isolation: Each processing pipeline gets its own consumer group

  • Stream Analytics: consumer group cg-asa
  • Flink: consumer group cg-flink
  • Archiver: consumer group cg-capture
  • Each reads independently without interference

Auto-inflate: Starts at provisioned TUs, scales automatically to 40 TUs. For higher throughput, use Event Hubs Dedicated clusters (up to 100 CUs).

Capture: Auto-archive all events to Data Lake in Avro format

  • Configurable: every 5 minutes or every 300 MB, whichever comes first
  • Zero-code backup — enables full replay from Data Lake if reprocessing needed

Partition count tradeoffs:

Partitions
Pros
Cons
32
Simple management, fast rebalance
Limited parallelism, hot partitions
128
Good parallelism for 1M events/sec
Moderate rebalance time (~30s)
256
Maximum throughput, fine-grained
Slow rebalance, more consumer overhead

Deep Dive 2: Windowed Aggregations and State Management

Azure Stream Analytics (SQL-like syntax for simple aggregations):

SELECT
sensor_id,
AVG(temperature) AS avg_temp,
MAX(temperature) AS max_temp,
COUNT(*) AS reading_count,
System.Timestamp() AS window_end
FROM
IoTEvents TIMESTAMP BY event_time
GROUP BY
sensor_id,
TumblingWindow(minute, 5)
HAVING
AVG(temperature) > 80

Window types:

Type
Description
Use Case
Tumbling
Fixed, non-overlapping (every 5 min)
Periodic reporting
Hopping
Overlapping (5 min window, every 1 min)
Smoothed metrics
Sliding
Triggered by events within duration
Real-time alerts
Session
Activity-based (gap = end of session)
User session analytics

Flink checkpointing (for complex stateful processing):

  • Algorithm: Chandy-Lamport distributed snapshots
  • Checkpoint interval: 30 seconds
  • State backend: RocksDB on Azure Blob Storage (supports TBs of state)
  • Exactly-once: checkpoint barriers align across operators; on failure, rewind to last checkpoint

Watermark algorithm:

W(t) = max(event_time seen across partition) - tolerance
If event.event_time < W(t): mark as "late"

Watermark tolerance tradeoffs:

Tolerance
Accuracy
Latency
State Size
1 minute
Lower (drops more late events)
Low
Small
5 minutes
Good balance
Moderate
Moderate
15 minutes
High (catches most late events)
High
Large windows in memory

Deep Dive 3: Materialized View Serving (Cosmos DB)

The challenge: Serve pre-computed aggregations with single-digit-ms latency for dashboards and APIs.

Partition key design: source_id as partition key

  • Time-series queries: point-read by source_id, then range filter on window_end
  • Cross-source queries: use Cosmos DB cross-partition queries (slower, for analytics only)

TTL: Auto-expire old aggregations — set TTL to 90 days (7,776,000 seconds)

  • Old windows automatically deleted; no manual cleanup needed

Change feed: Cosmos DB change feed triggers downstream processing

  • Alert evaluator subscribes to change feed → evaluates thresholds on each new aggregation
  • Enables reactive alerting without polling

Consistency level tradeoffs:

Level
Latency
Guarantee
Use Case
Strong
Higher (~10ms)
Linearizable reads
Billing/financial metrics
Bounded Staleness
Medium (~5ms, max 5s lag)
Reads within staleness window
Default for dashboards
Eventual
Lowest (~2ms)
No ordering guarantee
Non-critical analytics

Deep Dive 4: Alert Engine

Architecture:

Cosmos DB Change Feed → Alert Evaluator → Notification Service → PagerDuty/Teams/Email

Alert types:

1. Threshold: metric > static value (e.g., avg_temp > 80°)

2. Rate change: metric increased 5x over baseline in window

3. Anomaly detection: 7-day rolling baseline, z-score > 3 triggers alert

Deduplication: Don’t flood on-call with repeated alerts

  • Group alerts by rule_id + source_id
  • Configurable cooldown (default 15 minutes)
  • State: last_triggered_at per (rule, source) in Cosmos DB

Fan-out: Azure Monitor → PagerDuty, Teams webhook, email

  • Each channel independently configurable per alert rule
  • Teams: rich card with metric graph + runbook link

Evaluation location tradeoffs:

Approach
Pros
Cons
In-stream (Flink operator)
Lowest latency, access to raw events
Tightly coupled to processing pipeline
Sidecar (poll Cosmos DB)
Decoupled, simple to deploy
Higher latency (~10s), polling overhead

Our choice: Change-feed-based evaluation — decoupled like sidecar but reactive like in-stream. Alert evaluator subscribes to Cosmos DB change feed, evaluates rules on each new aggregation.

7

Bottlenecks & tradeoffs

Bottlenecks

1. Event Hubs partition hot spots: Skewed partition key → one partition overloaded

  • Mitigation: Hash-based partitioning; monitor per-partition throughput; split hot keys

1. Flink state growth for long windows: 15-minute windows with 10M sources = TBs of state

  • Mitigation: RocksDB state backend with incremental checkpoints; TTL on state entries

1. Power BI push dataset rate limits: 1M rows/hour per dataset

  • Mitigation: Pre-aggregate before pushing (send window results, not raw events); use DirectQuery for high-volume

1. Late event storm after connectivity restoration: Millions of buffered events arrive simultaneously

  • Mitigation: Backpressure from Flink; rate-limit late correction pipeline; Event Hubs absorbs burst

Key Tradeoffs

Decision
Option A
Option B
Choice
Stream processor
Azure Stream Analytics (managed, SQL)
Flink (flexible, complex)
ASA for simple, Flink for complex
Event format
JSON (readable)
Avro (compact, schema)
Avro with JSON fallback
Exactly-once
At-least-once + idempotent sinks
Exactly-once (Flink checkpoints)
At-least-once + idempotent for most; exactly-once for billing
Late events
Drop (simple)
Process in correction pipeline
Process (accuracy > simplicity)
Dashboard update
1 second (responsive, expensive)
5 seconds (efficient)
5 seconds for most, 1 second for ops

Monitoring & Alerting

  • Event Hubs throughput: Alert if > 80% of provisioned TUs
  • Consumer lag: Alert if any consumer group > 30 seconds behind
  • Late event rate: Alert if > 5% of events arrive late (indicates clock skew or connectivity issues)
  • Flink checkpoint duration: Alert if > 60 seconds (risk of data loss on failure)
  • Dashboard refresh latency: Alert if Power BI tile stale > 30 seconds
  • Stream Analytics job failures: Auto-restart + alert after 3 consecutive failures
  • Cosmos DB RU consumption: Alert if > 70% of provisioned RU/s

Failure Modes

  • Event Hubs partition leader failover: Automatic rebalance, ~30s disruption; producers buffer locally
  • Flink TaskManager crash: Job restarts from last checkpoint; exactly-once semantics preserved
  • Cosmos DB throttling (429s): Retry with exponential backoff; autoscale RU/s kicks in within minutes
  • Power BI push failure: Events queued in Cosmos DB; dashboard shows stale-data indicator
  • IoT Hub regional outage: Devices buffer locally; reconnect to secondary region via Azure DPS (Device Provisioning Service)
★

Rubric — Senior vs Staff

Dimension
Senior signal
Staff signal
Stream processing
Picks one engine and runs windowed aggregations.
Separates managed SQL (simple) from Flink (complex stateful); justifies the per-pipeline split.
Late & out-of-order events
Mentions watermarks and a tolerance window.
Designs a correction pipeline and reasons about tolerance vs latency vs state size.
Processing guarantees
Knows at-least-once vs exactly-once.
At-least-once with idempotent sinks by default; exactly-once via checkpoints only for billing.
Ingestion & partitioning
Partitions by source_id to preserve ordering.
Mitigates hot partitions, sizes partition count against rebalance, isolates tenants and consumer groups.
Serving & alerting
Serves aggregations from a fast store and polls for thresholds.
Change-feed-driven evaluation with dedup/cooldown; tunes consistency level per metric criticality.

Related questions

★ MORE WALKTHROUGHS

Want more breakdowns like this?

Join free early access for upcoming RAG, LLM eval, agents, and AI infrastructure walkthroughs.

Join Free Early Access →