← Back to all questions
AI System DesignRAGEvaluationStaff
Design a RAG Customer Support System
Answer support questions from a knowledge base with grounded, low-hallucination retrieval.
100% free · No login required
WHAT THIS QUESTION TESTS
·Chunking, embeddings, and retrieval quality
·Grounding and hallucination control
·Latency budget and eval pipelines
★ STAFF-LEVEL SIGNALS
★Frames retrieval quality as the real bottleneck and measures it (recall@k, groundedness) rather than by vibes.
★Designs an eval pipeline: a golden Q&A set, offline scoring, and online feedback capture.
★Reasons about chunking / reranking tradeoffs and their effect on latency and answer quality.
★Sets a hard latency budget and chooses where to spend it (rerank vs a bigger model).
1
The prompt
Design a retrieval-augmented support assistant that answers customer questions from a company’s knowledge base. Answers must be grounded in real documents, keep hallucinations low, respond within a tight latency budget, and fall back safely (to a human or a “not found” response) when retrieval is weak or confidence is low.
2
Senior-level outline
- Clarify requirements: corpus size, freshness, latency target, and tolerance for wrong answers.
- Pipeline: ingest → chunk → embed → vector index → retrieve top-k → (rerank) → prompt LLM with context.
- Choose chunking and an embedding model; store vectors in a vector DB with metadata filters.
- Ground the model: instruct it to answer only from retrieved context and cite sources.
- Add a fallback: if top-k similarity is low, say “I don’t know” or route to a human.
- Cache embeddings and frequent answers; keep the index fresh as documents change.
3
Common mistakes
- Treating it as “just call an LLM” and skipping retrieval quality entirely.
- Dumping huge unstructured chunks so the model gets noisy, irrelevant context.
- No grounding instruction or citations, so the model invents answers.
- No evaluation plan — unable to say whether a change made quality better or worse.
- Ignoring the low-confidence path, so the system confidently answers when it shouldn’t.
◇
AI engineering connection
RAG is the canonical AI engineering system design question — it’s the architecture behind most real LLM apps. Every instinct it tests (retrieval quality, grounding, latency budgeting, evaluation, and safe fallback) is exactly what AI Engineer and LLM App Engineer interviews look for. If you can reason about a support RAG system end to end, you can reason about most production LLM products.
★
Rubric — Senior vs Staff
Dimension
Senior signal
Staff signal
Requirements
Captures corpus, latency, accuracy needs
Quantifies the cost of a wrong answer, sets a quality bar
Retrieval
Chunk + embed + vector search
Tunes chunking / reranking, measures recall@k
Grounding
Answers from context, cites
Confidence thresholds and an abstention strategy
Latency
Notes the budget
Allocates the budget across stages deliberately
Evaluation
Mentions testing
Offline golden set + online feedback loop
Reliability
Has a fallback
Handles stale docs, low confidence, human handoff
Ownership
Mentions monitoring
Owns groundedness drift, cost, and eval gates
★ MORE WALKTHROUGHS
Want more breakdowns like this?
Join free early access for upcoming RAG, LLM eval, agents, and AI infrastructure walkthroughs.