Design a Distributed Logging & Monitoring System

Framing statement:

“Before I dive in, let me frame how I’d approach this. A multi-tenant observability platform at AWS scale — 100K+ servers, 10TB/day of logs, 100B metric data points/day — spans multiple organizations: an agent/collection team owning the per-host agent and data shipping, an ingestion pipeline team owning Kinesis and Flink processing, a storage team owning OpenSearch and S3 lifecycle, a query/analytics team owning the search and dashboard experience, an alerting/anomaly team owning real-time detection, and a platform/tenancy team owning namespace isolation and chargeback. The central architectural challenge is sub-minute alerting while keeping per-GB costs below commodity thresholds — the alerting path must be decoupled from the query path so that a slow OpenSearch cluster never delays a critical page. I’ll focus primarily on the ingestion pipeline and tiered storage architecture, then deep-dive into multi-tenant isolation and anomaly detection.”

Phased delivery:

Team ownership:

Key ambiguity I’d challenge:

“When we say ’monitoring’ — are we including distributed tracing, or only logs and metrics? Tracing adds a fundamentally different data model (span trees vs time-series vs full-text). I’ll design for logs + metrics core in Phase 1-2 and add tracing integration in Phase 3, where we ingest X-Ray spans and correlate them with logs and metrics via trace_id.”

Functional Requirements (Phase 1-2)

• ·FR1: Collect logs from 100K+ servers with configurable sampling/filtering at agent level
• ·FR2: Ingest structured metrics (counters, gauges, histograms) at 100B data points/day
• ·FR3: Full-text search on logs with sub-5s latency for recent data (last 24h)
• ·FR4: Real-time alerting on metrics (threshold + anomaly) with sub-60s detection
• ·FR5: Multi-tenant isolation — each team sees only their own data, enforced at query and storage level
• ·FR6: Configurable retention policies per team/service (7 days to 7 years)
• ·FR7: Dashboards with pre-aggregated metrics for common operational views (p50/p99 latency, error rate, throughput)

Non-Functional Requirements

Organizational Context

• ·Existing system: Evolution from CloudWatch + per-team bespoke solutions (ELK stacks, custom Kinesis pipelines, Grafana+InfluxDB). No single platform covers all teams — duplication of effort and inconsistent alerting quality.
• ·Regulatory: SOC2/PCI logs must be immutable (WORM — Write Once Read Many). Audit logs require 7-year cold retention with tamper-proof verification (S3 Object Lock).
• ·LP Connection: Dive Deep — engineers need deep observability into every service to diagnose issues. Frugality — tiered storage and chargeback ensure cost efficiency at scale.
• ·Timeline: Phase 1 in 3 months with Agent + Ingestion Pipeline teams (12-15 engineers).

Out of Scope (Explicitly)

• ·APM code-level profiling (Phase 3)
• ·Distributed tracing storage and span indexing (Phase 3)
• ·External customer-facing CloudWatch (separate product team)
• ·Log content — we store and index; we do not interpret application-specific semantics

Scale Numbers

Storage Estimation

Cost Estimation

Architectural implication: Storage is the dominant cost — Frugality LP demands tiered architecture. Keeping all logs in hot OpenSearch for 30 days would cost ~$650K/month. Hot/warm/cold tiering reduces this by ~75%. Chargeback makes teams accountable for their log volume, creating natural incentive to reduce log spam and set appropriate retention.

External API (REST)

Inter-Service Contracts (gRPC)

API Governance

• ·Versioning: URI versioning (/api/v1/), Protobuf for all internal gRPC services
• ·Rate limiting: Per-namespace ingestion rate limit (token bucket, configurable), per-namespace query concurrency limit (default 10 concurrent queries)
• ·Schema registry: Protobuf schemas in internal registry, backward-compatible evolution enforced by CI
• ·Deprecation: External API versions supported minimum 12 months; internal gRPC services require 3-month migration window

Core Entities

Data Ownership Boundaries

Storage Technology Choices

Data Lifecycle

Configurable per namespace: security/audit logs retain hot 90 days, cold 7 years with S3 Object Lock (WORM). Development service logs retain hot 24h, warm 7 days, no cold.

Architecture Diagram

Service Boundaries with Ownership

Sync vs Async Communication

Failure Domain Boundaries

Deep Dive 1: Log Agent Architecture

Tradeoff: Push vs Pull Ingestion

Recommendation: Push model. Each host runs a CloudWatch Agent that tails log files and pushes to Kinesis. Agent-side filtering means we never pay to transport DEBUG logs we will discard.

Agent Protocol Detail:

Deep Dive 2: Ingestion Pipeline (Kinesis + Flink)

Tradeoff: Kinesis vs MSK (Managed Kafka) vs Self-Managed Kafka

Recommendation: Kinesis Data Streams. The Frugality LP favors lower operational cost over raw $/event savings. Native Flink connector with enhanced fan-out provides dedicated throughput per consumer. 7-day retention gives us ample replay buffer.

Flink Pipeline Stages:

Deep Dive 3: Multi-Tenant Isolation

Tradeoff: Shared Cluster vs Per-Tenant Cluster vs Hybrid

Recommendation: Hybrid model. Shared OpenSearch cluster for ~1,990 standard namespaces. Dedicated clusters for the top-10 highest-volume teams (which generate ~60% of total volume). This provides cost efficiency (Frugality LP) with strong isolation for the teams most likely to cause noisy-neighbor issues.

Namespace Implementation:

Chargeback Model (Frugality LP):

Dashboard per namespace showing: daily cost breakdown, ingestion trend, top-10 chattiest services, cost optimization recommendations (“Enable DEBUG sampling to save $X/month”, “Reduce retention from 30d to 14d warm to save $Y/month”).

Deep Dive 4: Data Tiering Architecture

Tradeoff: Age-Based vs Access-Frequency vs Hybrid Transition

Recommendation: Age-based with manual override. Default policy transitions by age. During incidents, operators can “promote” a cold time range back to warm for faster investigation (takes ~5 min for Athena-to-OpenSearch rehydration).

Lifecycle Protocol:

Deep Dive 5: Anomaly Detection on Metrics

Tradeoff: Statistical (Z-Score/EWMA) vs ML (Isolation Forest/LSTM)

Recommendation: Statistical methods for Phase 1 (EWMA + Z-Score) — immediate value, interpretable alerts, low cost. ML augmentation in Phase 2 — isolation forest for multivariate anomalies, trained offline, inference at edge.

Detection Protocol:

Context

Many teams across the organization currently run bespoke observability solutions: per-team ELK stacks, custom Kinesis pipelines to CloudWatch, self-managed Grafana+InfluxDB. These solutions are expensive to maintain, create inconsistent alerting quality, and make cross-team debugging nearly impossible. The rollout must absorb teams gradually without disrupting their existing workflows.

Phase 1: Internal Dogfood (Weeks 1-8)

• ·Deploy platform for observability team itself — eat our own dog food
• ·5 pilot teams (volunteers from high-pain teams), 500 servers
• ·Agent deployment via SSM Run Command (zero-touch installation)
• ·Dual-write: agent ships to both new platform AND existing ELK/CloudWatch
• ·Success criteria: ingestion latency p99 < 30s, zero data loss, 5 teams confirm parity
• ·Weekly feedback sessions with pilot teams

Phase 2: High-Value Teams (Weeks 9-20)

• ·Top-20 highest-volume teams (~60% of total log volume)
• ·Chargeback reporting active (visibility only, not charged yet)
• ·Dedicated OpenSearch cluster for top-3 teams (noisy neighbor prevention)
• ·Per-team feature flag: can route each team's traffic independently
• ·Migration tooling: import Kibana dashboards → platform dashboards (automated converter)
• ·Success criteria: 20 teams migrated, chargeback reports accurate to 99%

Phase 3: Broad Rollout (Weeks 21-36)

• ·Self-service onboarding portal: create namespace, install agent, configure retention
• ·500 teams, 50K servers
• ·One-click migration from ELK: import dashboards, alert rules, saved searches
• ·CloudWatch Logs routing: subscription filter sends to Kinesis (for teams that cannot change agent)
• ·Chargeback active in team cost reporting (charged to team's AWS account)
• ·Success criteria: 80% of teams migrated, self-service onboarding < 30 minutes

Phase 4: Mandatory Adoption (Months 9-15)

• ·VP mandate: all teams must migrate by Month 15
• ·Decommission per-team ELK stacks (save ~$200K/month in duplicated infrastructure)
• ·Full chargeback in team cost reporting — visible in monthly business reviews
• ·Platform team provides migration office hours and dedicated support
• ·Success criteria: 100% migration, per-team ELK decommissioned

Rollback Plan

Bottleneck Analysis

Observability (RED Method)

Critical Alerts (Meta-Monitoring)

1. Ingestion lag > 5 minutes — Kinesis iterator age (GetRecords.IteratorAgeMilliseconds) exceeds 300s → page on-call

2. OpenSearch cluster health RED/YELLOW > 5 min — shard allocation failure, potential data loss risk → page on-call

3. Alert evaluation latency p99 > 30s — alerting pipeline degraded, critical alerts may be delayed → page on-call

4. Namespace ingestion exceeds quota by 2x — runaway logging, potential noisy neighbor → notify namespace owner + platform on-call

5. Agent deployment coverage < 99% — hosts missing agent, blind spots in observability → alert platform team

Distributed Tracing (Own Platform)

• ·Trace from log emission → agent batch → Kinesis PutRecords → Flink processing → OpenSearch bulk index
• ·Each stage adds span with timing: agent.batch_ms, kinesis.put_ms, flink.process_ms, opensearch.index_ms
• ·Own services instrumented with X-Ray — traces visible in the platform itself (dogfooding)
• ·End-to-end latency breakdown enables targeted optimization (identify which stage contributes most to p99)

2-3 Year Evolution

Year 1: Core Platform

• ·Core logging + metrics with tiered storage operational
• ·80% of teams migrated from bespoke solutions
• ·Statistical anomaly detection live
• ·Chargeback active, driving 20% volume reduction through team accountability
• ·Single-region, with Kinesis cross-region replication for DR

Year 2: Intelligence

• ·ML anomaly detection (isolation forest + LSTM for seasonality)
• ·Automated root cause analysis — correlate metrics, logs, and deployment events
• ·Distributed tracing integration (X-Ray span ingestion, trace_id correlation with logs)
• ·Log pattern detection — automatically cluster similar error messages (Dive Deep LP)
• ·Cross-namespace correlation — detect cascading failures across service boundaries

Year 3: Unified Platform

• ·Natural language log search — "show me errors from payment service yesterday during the latency spike"
• ·Predictive alerting — forecast metric trends, alert before threshold breach
• ·Unified observability: logs + metrics + traces + profiling in single pane of glass
• ·Self-healing: anomaly detection triggers automated runbooks (SSM Automation)
• ·Potential external offering — productize as managed service for AWS customers (Think Big LP)

Cost Optimization Roadmap

The key staff-level insights in this design:

1. Tiered storage as the central cost optimization — hot/warm/cold architecture reduces storage costs by 75% vs all-hot, directly aligned with Frugality LP. Age-based transitions with manual override for incidents.

2. Multi-tenant chargeback creates natural cost pressure — namespaces with per-team quotas and cost attribution incentivize teams to log at appropriate levels, reducing volume 20% without mandating specific behaviors.

3. Kinesis + Flink exactly-once pipeline — Kinesis as durable buffer (7-day retention) decouples collection from processing. Flink checkpointing to S3 ensures exactly-once semantics. Pre-aggregation in Flink reduces downstream write volume 10x.

4. Separate failure domains for ingestion vs alerting — alerting runs as a dedicated Flink CEP job, independent from the OpenSearch query path. An OpenSearch outage does not delay critical alerts.

5. Migration strategy for diverse existing solutions — phased rollout from pilot → high-value → broad → mandatory, with per-team feature flags, dual-write validation, and ELK dashboard import tooling. This is where most “platform” projects fail.

6. 3-year evolution from logs+metrics → unified observability — statistical anomaly detection (Year 1) → ML root cause analysis + tracing (Year 2) → NL search + predictive alerting + self-healing (Year 3). Each phase delivers standalone value while building toward the unified platform.