StaffSearchRanking

Design a Product Search & Ranking System

A Staff-level walkthrough of an e-commerce product search system that ranks 600M+ listings under a 200ms budget — trading recall against precision through a two-stage retrieval-and-ranking funnel, then blending organic results with sponsored placements. It weighs multi-objective scoring across relevance, purchase likelihood, and business value, near-real-time index freshness, and a cross-team ranking-model migration. Follow it top to bottom, or jump to any step.

Level: Staff
Category: Search · Ranking
Interview time: 60 min

100% free · No login required

WHAT THIS QUESTION TESTS

·Two-stage funnel: BM25 + ANN recall, then neural re-ranking

·Multi-objective scoring: relevance, conversion, business value

·Sponsored blending behind a relevance gate, with organic fallback

·Near-real-time index freshness for price and stock changes

★ STAFF-LEVEL SIGNALS

★Justifies the funnel by the math — scoring 600M docs per query can't fit 200ms

★Isolates failure domains so ad or ML outages degrade rather than break search

★Owns a cross-team ranking-model migration via shadow, interleaving, and progressive rollout

★Names ML inference as the dominant cost; uses distillation and result caching as levers

Scope & ambiguity

Framing statement:

“Before I dive in, let me frame how I’d approach this. Amazon’s product search spans multiple organizations — a search infrastructure team owning the index, a ranking/ML team owning relevance, an ads team owning sponsored placements, a catalog team owning product data, and a frontend team owning the search experience. The key architectural challenge is building a multi-objective ranking system that balances relevance, purchase likelihood, and monetization — all within a 200ms latency budget across 600M+ products and billions of daily queries. I’ll focus on the indexing pipeline, query understanding, ranking model (A9/A10), and sponsored product integration, and propose a phased approach.”

Phased delivery:

Phase

Timeline

Scope

Phase 1

0-3 months

Core keyword search with BM25 retrieval, basic category filtering, single-region deployment, p99 < 300ms

Phase 2

3-9 months

Semantic search via embeddings, personalized ranking, real-time index updates (<5 min freshness), multi-region, p99 < 200ms

Phase 3

9-18 months

Conversational search (Rufus integration), visual search, voice search (Alexa), cross-marketplace unification, anticipatory search

Team ownership:

Team

Ownership

Search Infrastructure Team

Inverted index, OpenSearch clusters, index pipeline, query routing

Ranking & Relevance Team (A9/A10)

ML ranking models, feature engineering, offline evaluation, A/B testing

Query Understanding Team

Tokenization, spell correction, intent classification, query expansion

Requirements

Functional Requirements (Phase 1-2)

FR1: Users can search the product catalog by keywords, returning ranked results within a latency budget
FR2: Results are ranked by a multi-objective function combining relevance, purchase likelihood, and business value
FR3: Search supports filtering by category, price range, brand, ratings, Prime eligibility, and other facets
FR4: New/updated products (price, stock, new listings) are searchable within 5 minutes
FR5: Sponsored products appear within organic results, subject to relevance constraints
FR6: Search is personalized based on user history, location, and Prime membership status

Non-Functional Requirements

Requirement

Target

End-to-end latency

p50 < 100ms, p99 < 200ms

Catalog size

600M+ active listings (350M+ unique products, rest are marketplace variants)

Search QPS

~150K sustained, ~500K peak (Prime Day)

Index freshness

Price/stock changes < 1 min, new listings < 5 min

Availability

99.99% (search downtime = revenue loss at ~$4,700/sec)

Relevance quality

Measured by NDCG@10, search-to-purchase conversion rate

Geographic coverage

20+ marketplaces, 200+ countries, localized results

Organizational Context

Existing system: A9/A10 search engine is mature — this is an evolution, not greenfield. The A10 generation added emphasis on organic engagement, external traffic signals, and seller authority over pure sales velocity.
Regulatory: GDPR (EU), CCPA (California), data residency requirements for marketplace-specific data, FTC advertising disclosure rules for sponsored results.
Infrastructure: Built on AWS — DynamoDB (151M req/sec peak on Prime Day 2025), SQS (166M msg/sec peak), Aurora, OpenSearch, SageMaker for ML inference, Kinesis for streaming.
Timeline: Phase 1 in 3 months with Search Infrastructure + Ranking teams (12-15 engineers).

Out of Scope (Explicitly)

Alexa voice search interface (Phase 3)
Visual search / search-by-image (Phase 3)
Rufus conversational AI search (Phase 3)
Seller-facing search analytics dashboard
Amazon Fresh / Whole Foods specialized search

Back-of-envelope estimation

Scale Numbers

Metric

Value

Active product listings

~600M (350M unique products + marketplace variants)

Search queries per day

~3-5B (across all marketplaces)

Peak QPS (Prime Day)

~500K

Sustained QPS (normal)

~150K

Unique searchers per day

~200M

Avg queries per session

~5-8

Index updates per day

~100M (price, stock, new listings, delistings)

Storage Estimation

Data

Calculation

Result

Inverted index (text)

600M products x 2KB avg per doc

~1.2TB

Embedding index (semantic)

600M x 256-dim x 4B per float

~600GB

Product metadata (DynamoDB)

600M x 5KB avg

~3TB

Query logs (30 days)

5B/day x 30 x 500B avg

~75TB

ML feature store

600M products x 200 features x 8B

~960GB

Total hot storage

~6TB index + ~80TB logs

Cost Estimation

Resource

Calculation

Monthly Cost

OpenSearch clusters (index)

200 nodes x r6g.4xlarge ($1.00/hr)

~$146K

ML inference (SageMaker)

500 endpoints x ml.g5.xlarge ($1.00/hr) for ranking

~$365K

DynamoDB (product metadata)

150K RCU sustained + burst

~$200K

Kinesis (real-time pipeline)

100 shards x $0.015/hr + PUT costs

~$50K

S3 + data transfer

Query logs, model artifacts, cross-region

~$100K

Total

~$860K/month

Architectural implication: ML inference for ranking is the dominant cost — each query invokes the ranking model on 1000+ candidates. We need a two-stage retrieval/ranking architecture to limit the number of candidates scored by the expensive neural model. Batch inference and model distillation (teacher → student) are critical cost levers.

API design

External API (Client-Facing)

# Product Search

GET /api/v1/search?q={query}&category={cat}&page={p}&size={n}

&price_min={min}&price_max={max}&brand={b}&prime_only={bool}

&sort={relevance|price_asc|price_desc|rating|newest}

Response: { results: [Product], facets: {Facet}, total_count, spell_correction, query_intent }

# Autocomplete / Suggestions

GET /api/v1/suggest?q={prefix}&marketplace={m}

Response: { suggestions: [{ text, category_hint, result_count }] }

# Product Detail (search click-through)

GET /api/v1/products/{asin}

Response: { product: Product, related_searches: [...] }

# Search Feedback (implicit signals)

POST /api/v1/search/events

Body: { query_id, event_type: "click"|"add_to_cart"|"purchase", asin, position, timestamp }

Inter-Service Contracts

SERVICE: QueryUnderstanding

OWNER: Query Understanding Team

PROTOCOL: gRPC

RPC ParseQuery(ParseQueryRequest) returns (ParseQueryResponse)

- Input: raw query string, marketplace, user context

- Output: tokens, spell_correction, intent, category_hint, price_filter, brand_filter, expanded_terms

- Latency SLA: p99 < 15ms

- Availability SLA: 99.99%

EVENTS PUBLISHED:

Topic: search.query-understanding

Schema: QueryParseResult (Protobuf)

Events: QUERY_PARSED, SPELL_CORRECTED, INTENT_CLASSIFIED

Partition key: query_hash

Retention: 7 days

SERVICE: SearchRetrieval

OWNER: Search Infrastructure Team

PROTOCOL: gRPC

RPC RetrieveCandidates(RetrievalRequest) returns (RetrievalResponse)

- Input: parsed query, filters, top_k (default 1000)

- Output: candidate ASINs with initial BM25 scores, facet counts

- Latency SLA: p99 < 30ms

- Availability SLA: 99.99%

SERVICE: RankingService

OWNER: Ranking & Relevance Team

PROTOCOL: gRPC

RPC RankCandidates(RankRequest) returns (RankResponse)

- Input: candidate ASINs (up to 1000), query context, user context

- Output: ranked ASINs with final scores, explanation metadata

- Latency SLA: p99 < 50ms (includes ML inference)

- Availability SLA: 99.99%

SERVICE: SponsoredProductsService

OWNER: Sponsored Products Team

PROTOCOL: gRPC

RPC GetSponsoredPlacements(PlacementRequest) returns (PlacementResponse)

- Input: query, organic results (top 50), available ad slots

- Output: sponsored ASINs with placement positions, bid amounts

- Latency SLA: p99 < 20ms

- Availability SLA: 99.95% (graceful degradation: show organic-only if ads fail)

EVENTS PUBLISHED:

Topic: ads.impressions

Schema: AdImpression (Protobuf)

Events: AD_SERVED, AD_CLICKED, AD_CONVERTED

Partition key: campaign_id

Retention: 90 days

API Governance

Versioning: URI versioning for external (/v1/), header versioning for internal gRPC services
Schema registry: Protobuf for all inter-service and event schemas, enforced via CI/CD pipeline
Rate limiting: Per-user: 100 searches/min, per-IP: 500/min (bot protection), per-seller-API: 1000/min
Deprecation policy: Minimum 6-month sunset window for external API versions; internal services require 3-month migration window

Data model

Core Entities

Entity: Product (ASIN)

- asin: VARCHAR(10) (PK) — Amazon Standard Identification Number

- marketplace_id: INT (partition key for multi-marketplace)

- title: TEXT

- brand: VARCHAR(256)

- category_path: VARCHAR(512) — "Electronics > Audio > Headphones"

- bullet_points: TEXT[] — up to 5 feature bullets

- description: TEXT

- price: DECIMAL(10,2)

- currency: VARCHAR(3)

- images: JSON — [{ url, alt_text, position }]

- attributes: JSON — { color, size, weight, ... }

- created_at: TIMESTAMP

- updated_at: TIMESTAMP

Entity: ProductSearchDocument (denormalized for index)

- asin: VARCHAR(10) (PK)

- marketplace_id: INT

- title_tokens: TEXT — pre-tokenized, normalized

- backend_keywords: TEXT — seller-provided hidden search terms

- category_id: INT

- brand_normalized: VARCHAR(256)

- price_cents: BIGINT — for range filtering

- avg_rating: FLOAT

- review_count: INT

- sales_rank: INT — within category

- prime_eligible: BOOLEAN

- fulfillment_type: ENUM (FBA, FBM, AMAZON)

- in_stock: BOOLEAN

- embedding_vector: FLOAT[256] — semantic embedding

- last_indexed_at: TIMESTAMP

Entity: SearchEvent (click-stream)

- event_id: UUID (PK)

- query_id: UUID

- session_id: UUID

- user_id: VARCHAR(64) — hashed

- query_text: TEXT

- asin: VARCHAR(10)

- position: INT

- event_type: ENUM (IMPRESSION, CLICK, ADD_TO_CART, PURCHASE)

- timestamp: TIMESTAMP

- marketplace_id: INT

Entity: SponsoredBid

- campaign_id: UUID

- asin: VARCHAR(10)

- keyword: VARCHAR(256)

- match_type: ENUM (EXACT, PHRASE, BROAD)

- bid_amount_cents: INT

- daily_budget_cents: BIGINT

- status: ENUM (ACTIVE, PAUSED, BUDGET_EXHAUSTED)

- relevance_score: FLOAT — pre-computed

- updated_at: TIMESTAMP

Entity: UserSearchProfile (personalization)

- user_id: VARCHAR(64) (PK)

- recent_categories: INT[] — last 30 days

- brand_affinities: JSON — { brand: score }

- price_sensitivity: FLOAT — 0.0 (luxury) to 1.0 (budget)

- prime_member: BOOLEAN

- marketplace_id: INT

- updated_at: TIMESTAMP

Data Ownership Boundaries

Data

Owner Service

Access Pattern for Others

Product catalog (ASIN)

Product Catalog Service

Sync gRPC (cached 5 min in local cache)

Search index documents

Search Infrastructure

Internal only — written by index pipeline

Search events / click-stream

Search Analytics Service

Async via Kinesis stream

High-level architecture

Architecture Diagram

[CloudFront CDN]

[Global Load Balancer]

+---------------+---------------+

| |

[US Region] [EU Region]

| |

[API Gateway] [API Gateway]

(Search Experience) (Search Experience)

| | | |

[Autocomplete] [Search Orchestrator] ... ...

(Suggest Svc) (Search Experience Team)

+------------+------------+------------+

| | | |

[Query Under- [Search [Ranking [Sponsored

standing] Retrieval] Service] Products]

(QU Team) (Infra Team) (A9/A10 Team) (Ads Team)

| | | |

| [OpenSearch [SageMaker [DynamoDB]

| Cluster] Endpoints] (bid store)

| (inverted (ML ranking

| + vector inference)

| index)

| |

+-------+-------+

[Index Pipeline] <--- [Kinesis] <--- [Product Catalog Service]

(Search Infra) (Catalog Team)

| |

[OpenSearch] [DynamoDB]

(bulk + real-time (source of truth)

index updates)

[Async Analytics Pipeline] --- [Kinesis] --- [S3 Data Lake]

(click-stream, model training, A/B test analysis)

[SageMaker Training] --- [Feature Store] --- [Model Registry]

Service Boundaries with Ownership

Service

Team

Data Store

Scaling

Consistency

SLA

API Gateway

Search Experience

—

Horizontal, per-region, auto-scaling

—

p99 < 5ms overhead

Search Orchestrator

Search Experience

Local cache (LRU)

Horizontal (stateless)

—

p99 < 200ms end-to-end

Query Understanding

DynamoDB (synonym/spell dictionaries)

Horizontal (stateless, model in memory)

Eventual (dictionary updates)

p99 < 15ms

Search Retrieval

Search Infrastructure

OpenSearch (inverted + vector index)

Horizontal sharding by category hash

Eventual (index lag < 5 min)

p99 < 30ms

Ranking Service

Ranking & Relevance

SageMaker endpoints + Feature Store

Auto-scaling endpoints, GPU inference

—

p99 < 50ms

Deep dives

WHERE STAFF IS WON

Core Decision: Two-Stage Retrieval + Ranking

This is the most consequential architectural decision for search at Amazon’s scale. Scoring all 600M products per query is impossible within 200ms. We need a funnel architecture.

Stage

Candidates

Latency Budget

Model Complexity

Purpose

Stage 0: Query Understanding

N/A

15ms

NLP models (spell, intent, expansion)

Parse and enrich the raw query

Stage 1: Retrieval

600M → 1,000

30ms

BM25 + ANN vector search

Fast recall — find all plausible candidates

Stage 2: Ranking

1,000 → 50

50ms

Deep neural network (GBDT + transformer)

Precision — score with full feature set

Stage 3: Blending

50 + ads → final SERP

10ms

Business rules + ad auction

Merge organic + sponsored, apply diversity rules

Why two-stage over single-stage: A single neural model scoring 600M products at ~0.1ms each would take 60,000 seconds. Two-stage reduces this to BM25 over the inverted index (O(query_terms) with posting list intersection) plus neural scoring of 1,000 candidates (~50ms on GPU). This is the standard approach used by Amazon, Google, and all large-scale search systems.

Query Understanding Pipeline

User query: "wireless noise canceling headphones under $100"

1. TOKENIZATION + NORMALIZATION:

- Lowercase, remove punctuation

- Tokens: [wireless, noise, canceling, headphones, under, $100]

2. SPELL CORRECTION:

- "headphoens" → "headphones" (edit distance + frequency model)

- "cancelling" accepted (variant spelling)

- Implementation: SymSpell algorithm with a 10M-term frequency dictionary

built from query logs. Backed by a transformer model for context-aware

correction ("appel" → "apple" for electronics, "appel" → "appel" for food).

3. QUERY PARSING:

- Product type: "headphones" (category intent)

- Attributes: "wireless" (connectivity), "noise canceling" (feature)

- Price constraint: "under $100" → price_filter < 100

- Brand detection: if "sony wireless headphones" → brand_filter = Sony

- Implementation: NER model (fine-tuned BERT) trained on 100M+ annotated queries.

4. QUERY EXPANSION:

- Synonyms: "noise canceling" → "noise cancelling", "ANC"

- Related terms: "wireless" → "bluetooth"

- Category mapping: "headphones" → Electronics > Audio > Headphones

- Implementation: embedding-based expansion — find terms with cosine

similarity > 0.85 in query embedding space. Curated synonym dictionary

as fallback.

5. INTENT CLASSIFICATION:

- Navigational: "apple airpods pro" → specific product (show product page)

- Informational: "best headphones 2024" → review-oriented results

- Transactional: "buy noise canceling headphones" → purchase-ready results

- Implementation: multi-class classifier (LightGBM) with features from

query structure, historical click patterns, and query frequency.

Indexing Pipeline

Real-time index updates:

[Product Catalog (DynamoDB Streams)] → [Kinesis] → [Index Updater Lambda]

→ Update inverted index (OpenSearch - text fields)

→ Update k-NN index (OpenSearch - embedding vectors)

→ Update facet aggregations (for filter counts)

→ Update price/stock fields (for freshness-critical data)

Inverted index structure (OpenSearch):

"wireless" → [ASIN_1 (tf-idf: 0.9), ASIN_2 (tf-idf: 0.8), ...]

"headphones" → [ASIN_2 (tf-idf: 0.95), ASIN_3 (tf-idf: 0.9), ...]

Per-product fields indexed with weights:

title (boost: 3.0), bullet_points (2.0), description (1.0),

brand (2.5), category (1.5), backend_keywords (hidden seller terms, 1.0)

Sharding strategy: 600M docs across ~200 shards

- Shard by category_hash for query locality (most queries target a category)

- Each shard: ~3M docs, ~6GB

- Replication factor: 2 (primary + 1 replica per shard)

Near-real-time freshness:

- Price/stock changes: reflected within 1 minute (critical for Buy Box accuracy)

- New listings: searchable within 5 minutes (index, compute embedding, extract features)

- Full re-index: weekly (for major algorithm updates or schema changes)

- Embedding re-computation: daily batch job via SageMaker Processing

Ranking Model (A9/A10)

Multi-objective ranking:

final_score = w1 * relevance + w2 * purchase_likelihood + w3 * business_value

Weights are tuned via online A/B tests — w1 ≈ 0.5, w2 ≈ 0.35, w3 ≈ 0.15

RELEVANCE FEATURES:

- Text match: BM25 score (title, bullets, description, brand)

- Semantic similarity: cosine(query_embedding, product_embedding)

- Category match: P(category | query) from intent classifier

- Attribute match: "wireless" in query → product.connectivity = "wireless"

- Query-title overlap: Jaccard similarity of token sets

PURCHASE LIKELIHOOD FEATURES:

- Conversion rate: historical purchases / impressions for this ASIN

- Price competitiveness: percentile rank vs similar products

- Review signal: log(review_count) * avg_rating (diminishing returns on count)

- Availability: in_stock (binary), delivery_speed (days), Prime eligible

- Seller quality: fulfillment defect rate, late shipment rate, return rate

- Sales velocity: units sold in last 7/30 days (A10 weights organic sales > ad-driven)

BUSINESS VALUE FEATURES:

- Margin contribution: Amazon retail vs 3P (commission %)

- Ad bid: sponsored product integration (see below)

- Prime membership value: Prime-eligible products boost for Prime members

- External traffic signal (A10-specific): products with off-Amazon demand rank higher

PERSONALIZATION FEATURES:

- User brand affinity: bought Sony 3x in last 6 months → boost Sony

- Price sensitivity: user's historical price distribution per category

- Category affinity: user frequently browses Audio → boost audio products

- Repeat purchase pattern: consumables user bought before → boost

Two-stage model architecture:

Stage 1 (Retrieval): BM25 + k-NN ANN search → top 1,000 candidates

- BM25 on inverted index: <10ms on OpenSearch

- k-NN on vector index: <15ms using HNSW algorithm

- Union of both result sets, deduplicated by ASIN

Stage 2 (Ranking): neural model with full feature set → top 50

- Model: GBDT (LightGBM) ensemble with a transformer cross-encoder re-ranker

- Input: ~200 features per (query, product) pair

- Inference: SageMaker endpoint, batched (1000 candidates), GPU-accelerated

- Latency: p99 < 50ms for 1000 candidates on ml.g5.xlarge

- Training: daily retrain on last 14 days of click/purchase data

- Evaluation: offline NDCG@10, online A/B test (purchase conversion rate)

Sponsored Products Integration

Sponsored results are ads that appear within organic search results:

AUCTION MECHANISM:

- Advertisers bid on keywords ("noise canceling headphones")

- Match types: EXACT ("noise canceling headphones"), PHRASE (contains phrase),

BROAD (semantically related)

- Ad rank = bid × relevance_score × expected_CTR

- Winner pays second-price: price needed to beat next bidder + $0.01

- Budget pacing: spread daily budget evenly to avoid morning exhaustion

PLACEMENT RULES:

- Top of SERP: 2 sponsored products (highest ad rank)

- Within results: position 4, 9, 14 can be sponsored (every ~5th slot)

- Bottom of SERP: 2-3 more sponsored products

- "Sponsored" label required (FTC compliance)

RELEVANCE GATING (critical constraint):

- Minimum relevance threshold (relevance_score > 0.6) to appear as ad

- If no relevant ads exist for query, show organic results only

- User trust metric: track user engagement with sponsored vs organic — if

sponsored CTR drops below 60% of organic CTR, tighten relevance gate

- Irrelevant ads hurt Customer Obsession and long-term search trust

BLENDING ALGORITHM:

1. Receive organic top-50 from Ranking Service

2. Receive eligible sponsored products from Sponsored Products Service

3. Insert sponsored products at designated positions IF they pass relevance gate

4. Apply diversity rules: no more than 2 products from same brand in top 10

5. Final SERP: interleaved organic + sponsored, max 48 results per page

Autocomplete and Query Suggestion

Autocomplete serves suggestions as the user types, with a <50ms latency budget:

DATA SOURCES:

- Query frequency logs: top 100M queries by frequency, updated daily

- Trending queries: last-24h query spikes (detects new product launches, events)

- Personal history: user's last 100 searches (stored client-side + server-side)

IMPLEMENTATION:

- Trie-based prefix index in memory (ElastiCache Redis)

- Each prefix maps to top-10 completions sorted by: frequency × recency × personalization

- Updated hourly from query log aggregation pipeline

- Fallback: if Redis unavailable, serve from local in-memory cache (updated every 5 min)

SAFETY FILTERING:

- Blocklist of offensive/dangerous completions

- Suppress suggestions for recalled products

- Legal review pipeline for trademark-sensitive completions

Multi-team rollout

Migration Strategy: A9 → A10 Ranking + Semantic Search

The system evolves from keyword-only BM25 search to a hybrid keyword + semantic retrieval with a more sophisticated A10 ranking model that incorporates external traffic and organic engagement signals.

Phase 1: Shadow Scoring (Weeks 1-6)

Deploy new A10 ranking model alongside existing A9 model
All queries scored by both models; A9 remains source of truth for production results
Log A10 scores for offline comparison: measure NDCG@10, conversion prediction accuracy
No user-facing impact; build confidence in A10 model quality
Team: Ranking & Relevance (4 engineers)

Phase 2: Interleaved Experiment (Weeks 7-12)

Run interleaved A/B test: for each query, randomly select half the result positions from A10, half from A9
Measure per-position click-through rate and purchase conversion
Simultaneously deploy semantic retrieval (k-NN) as an additional retrieval path in Stage 1
Gate: A10 must show statistically significant improvement (p < 0.05) on purchase conversion
Team: Ranking + Search Infrastructure (8 engineers)

Phase 3: Progressive Rollout (Weeks 13-20)

1% → 5% → 25% → 50% → 100% of traffic on A10 ranking
Rollout by marketplace: start with amazon.com (US), then .co.uk, .de, .co.jp
Monitor: conversion rate, revenue per search, customer complaints, seller escalations
Each marketplace requires localized model fine-tuning (language, buying patterns)
Team: All search teams (15+ engineers)

Phase 4: A9 Decommission (Weeks 21-28)

Migrate remaining long-tail marketplaces to A10
Decommission A9 scoring endpoints
Archive A9 model artifacts and training pipelines
Update on-call runbooks and dashboards

Rollback Plan

Stage

Rollback Action

Data Impact

Phase 1

Stop shadow scoring

None — A9 was always serving

Phase 2

Disable interleaving, route 100% to A9

None — both models were running

Phase 3

Feature flag: route traffic back to A9 endpoints

None — A9 still running in parallel until Phase 4

Phase 4

Cannot rollback (A9 decommissioned)

Must fix forward; keep A9 model artifacts archived 6 months for emergency re-deployment

Bottlenecks & evolution

Bottleneck Analysis

Component

Bottleneck

Mitigation

OpenSearch cluster

Hot shards for popular categories (Electronics, Clothing)

Shard by category_hash with pre-split for top-20 categories; dedicated hot-shard nodes with more RAM

ML ranking inference

GPU saturation during Prime Day (5x normal traffic)

Pre-warm SageMaker endpoints 24h before; auto-scaling with predictive scaling policy; fallback to CPU-based GBDT-only model (skip transformer re-ranker)

DynamoDB product reads

Throttling during catalog-wide price updates (Black Friday)

DAX caching layer, on-demand capacity mode, pre-provision for known events

Index pipeline (Kinesis)

Backpressure during bulk catalog updates (seller imports)

Kinesis enhanced fan-out, increase shard count, prioritize price/stock updates over metadata updates

Query Understanding

Cold start on new query patterns (trending products, viral events)

Pre-warm spell correction and intent models with trending Twitter/social data; fall back gracefully to keyword-only search

Observability

Key metrics (RED method):

Service

Rate

Errors

Duration

Search Orchestrator

Searches/sec by marketplace

4xx (bad query), 5xx (service failure), zero-result rate

End-to-end latency p50/p99

Query Understanding

Queries parsed/sec

Parse failures, spell correction false positives

Parse latency p50/p99

Search Retrieval

Retrievals/sec, candidates returned

Index errors, timeout rate, empty result sets

Retrieval latency by shard

Ranking Service

Candidates scored/sec

Inference failures, model version mismatch

Scoring latency per batch

Summary

The key Principal SDE (L7) insights in this design:

1. Two-stage retrieval + ranking funnel is the core architectural pattern — BM25 + ANN retrieval (1000 candidates in 30ms) followed by neural ranking (50 results in 50ms), enabling 200ms end-to-end on 600M products

2. Multi-objective ranking balances relevance, purchase likelihood, and business value — the weight tuning between these objectives is the most commercially impactful decision at Amazon

3. A9 → A10 migration via shadow scoring, interleaved experiments, and progressive rollout with per-marketplace fine-tuning — a Principal SDE owns the cross-team migration plan

4. Team boundaries follow service boundaries (Conway’s Law): Query Understanding, Search Infrastructure, Ranking/ML, Sponsored Products, Catalog, and Search Experience teams each own their service and data

5. Failure domain isolation ensures ads failures don’t break organic search, ML failures fall back to BM25, and index pipeline lag doesn’t affect serving

6. Cost model identifies ML inference as the dominant cost and proposes distillation + caching as primary levers — Frugality LP in action

7. 3-year evolution from keyword search → semantic + personalized → conversational + anticipatory + platform, positioning search as Amazon’s core commerce engine

★

Rubric — Senior vs Staff

Dimension

Senior signal

Staff signal

Retrieval architecture

Single index lookup returning ranked matches.

Two-stage funnel; quantifies why neural scoring of 600M docs can't fit a 200ms budget.

Ranking

Sorts by text relevance and rating.

Multi-objective score; tunes relevance/conversion/revenue weights via online A/B tests.

Index freshness

Batch re-index on a fixed schedule.

Streamed updates prioritizing price/stock (<1 min) over metadata; periodic full rebuild.

Monetization

Appends ads after the organic results.

Auction plus relevance gate and diversity rules; ads fail open to an organic-only page.

Rollout & failure handling

Flag-guarded deploy with a rollback path.

Shadow → interleaved → progressive per-marketplace; isolated failure domains degrade gracefully.

★ MORE WALKTHROUGHS

Want more breakdowns like this?

Join free early access for upcoming RAG, LLM eval, agents, and AI infrastructure walkthroughs.

Join Free Early Access →

Scope & ambiguity

Requirements

Back-of-envelope estimation

API design

Data model

High-level architecture

Deep dives

Multi-team rollout

Bottlenecks & evolution

Summary

Rubric — Senior vs Staff

Related questions

Want more breakdowns like this?