← Back to all questions
AI System DesignEvalObservabilityStaff
Design an LLM Evaluation Platform
Score model and prompt quality continuously with offline and online evaluation.
100% free · No login required
WHAT THIS QUESTION TESTS
·Offline vs online evaluation design
·Regression testing for prompts and models
·Production monitoring and feedback loops
★ STAFF-LEVEL SIGNALS
★Treats evaluation as a regression-testing system: a quality gate in CI for prompt / model changes.
★Reasons about metric validity — when to trust LLM-as-judge vs require human labels.
★Designs regression detection: compare a new version against the baseline before rollout.
★Closes the loop: online failures and human labels flow back into the offline golden set.
1
The prompt
Design a platform that continuously measures the quality of LLM-powered features. It must support offline evaluation against curated datasets and online evaluation on live traffic, detect regressions when prompts or models change, version prompts and models, incorporate human review, and gate releases on quality.
2
Senior-level outline
- Clarify the eval target: which feature, what “good” means, and acceptable cost / latency for evals.
- Offline: a versioned golden dataset scored by metrics and/or an LLM-as-judge on each change.
- Online: sample live traffic, log inputs / outputs, and collect implicit + explicit user feedback.
- Version prompts and models; attach every eval run to a specific version for comparison.
- Add human review for a sampled subset and for low-confidence or disagreement cases.
- Surface results in a dashboard with per-version scores and trends.
3
Common mistakes
- Relying only on offline accuracy on a tiny static set that doesn’t reflect production.
- Trusting an LLM judge blindly with no human calibration of the judge itself.
- No versioning, so you can’t tell which prompt or model caused a regression.
- Treating eval as a one-off rather than a continuous gate on every change.
- Collecting feedback but never feeding it back into the eval set.
◇
AI engineering connection
An eval platform is how AI teams keep quality from silently degrading in production. The skills it tests — regression testing for prompts and models, offline vs online measurement, human-in-the-loop review, and quality-gated rollouts — are the core of production AI reliability. AI Infrastructure and AI Engineer interviews increasingly center on this: not “can you call a model” but “can you prove it still works after every change.”
★
Rubric — Senior vs Staff
Dimension
Senior signal
Staff signal
Requirements
Defines a quality metric
Ties metrics to task value, sets a release bar
Offline eval
Golden set + metric
Versioned datasets, judge-validity reasoning
Online eval
Logs production traffic
Sampling, feedback capture, drift detection
Regression
Compares versions
CI quality gate before rollout, baseline diffing
Human review
Mentions human labels
Cost-aware sampling + judge calibration
Tradeoffs
Notes eval cost
Quantifies sampling vs cost vs signal
Ownership
Mentions a dashboard
Owns drift alerts, rollback, and the feedback loop
★ MORE WALKTHROUGHS
Want more breakdowns like this?
Join free early access for upcoming RAG, LLM eval, agents, and AI infrastructure walkthroughs.