← Back to all questions
AI System DesignStaffRAG
Design AI Copilot Infrastructure for Microsoft 365
A Staff-level walkthrough of the Microsoft 365 Copilot orchestration layer — routing 300M+ users through RAG grounding, multi-model inference, sandboxed plugins, and fail-closed safety within a 5-second TTFT budget, with strict tenant isolation and data residency.
100% free · No login required
WHAT THIS QUESTION TESTS
·RAG grounding via Semantic Index, not fine-tuning
·Multi-model routing for cost at scale
·Tenant isolation & data residency
·Plugin sandboxing & three-layer safety
★ STAFF-LEVEL SIGNALS
★Picks RAG over per-tenant fine-tuning at 30M tenants
★Model routing — a $12M/mo lever, quality held
★Tiered isolation: logical vs physical for regulated
★Safety fails closed; everything else degrades
0
Scope & ambiguity
Framing statement
“Microsoft 365 Copilot is a multi-team problem spanning the orchestration layer, RAG via the Semantic Index and Microsoft Graph, LLM inference through Azure OpenAI, plugin invocation, and responsible-AI filtering. I’d focus on the orchestration layer — how we route 300M+ M365 users through RAG grounding, LLM inference, plugin invocation, and safety filtering, all within a 5-second TTFT budget with strict tenant isolation and data-residency guarantees. I’ll propose a phased approach.”
Phased delivery
Phase
Timeline
Scope
Phase 1
0–4 months
Single-turn Copilot for Word & Outlook, RAG via Semantic Index, GPT-4 only, single region (US-West)
Phase 2
4–10 months
All M365 apps (Excel, Teams, PowerPoint), multi-model routing, first-party plugins, multi-region
Phase 3
10–18 months
Copilot agents (multi-step workflows), Copilot Studio, fine-tuned industry models, proactive suggestions
Team ownership
Team
Ownership
Copilot Orchestrator
Request lifecycle, prompt assembly, response streaming, model routing
Semantic Index
Vector embeddings, hybrid retrieval, security-trimmed search
Azure OpenAI Platform
Model hosting, inference endpoints, capacity management, token accounting
Plugin Platform
Plugin registry, manifest validation, sandboxed execution, OAuth brokering
Responsible AI
Pre/post safety filters, content classifiers, PII detection, grounding verification
Per-App Integration
Word, Excel, Outlook, Teams, PowerPoint — app-specific context extraction & UX
Microsoft Graph
Unified data API, change notifications, permission model, entity resolution
Compliance & Residency
Audit logging, Purview, EU Data Boundary, GCC High, GDPR/FedRAMP/HIPAA
Key ambiguity I’d challenge
“Two things. First, the latency contract: sub-2s TTFT for short-form (email reply, formula suggestion) vs sub-5s for long-form (document draft, meeting summary)? That decides whether we need a fast-path routing layer. Second, the isolation model: is logical isolation enough for commercial tenants, or do regulated customers (GCC High, EU sovereign) need physically separate inference pools? Cost and capacity planning differ by 5–10×.”
1
Requirements clarification
Functional requirements (Phase 1)
- FR1 — Invoke Copilot from any M365 app via inline prompt or side panel
- FR2 — Ground responses in the user’s own data via RAG (documents, emails, calendar, Teams chats from the Semantic Index)
- FR3 — All retrieved content respects M365 RBAC — the user only sees data they already have permission to access
- FR4 — Invoke plugins (first- and third-party) to fetch external data or take actions
- FR5 — Multi-turn conversations maintain context within a session
- FR6 — Every interaction is logged for compliance audit via Microsoft Purview
Non-functional requirements
Requirement
Target
TTFT (short-form, p50)
< 1.5s
TTFT (long-form, p99)
< 5s
Availability
99.9%
Grounding accuracy
> 90% of claims traceable to sources
Hallucination rate
< 5% (by grounding verification)
Cross-tenant data leakage
Zero tolerance (SEV-0)
EU data residency
All EU tenant data processed & stored in EU
Concurrent sessions
50K peak
Organizational context
- Business model: $30/user/month premium add-on to M365 E3/E5
- Existing infra: Azure OpenAI Service, Microsoft Graph, Semantic Index (all pre-existing)
- Regulatory: GDPR (EU residency), FedRAMP (US gov), HIPAA (healthcare), SOC 2
- Timeline: Phase 1 in 4 months — Orchestrator team (6 eng) + per-app integration engineers
Out of scope (explicitly)
- Azure OpenAI internals (model training, GPU cluster management, inference kernel)
- GitHub Copilot (separate product, separate infrastructure)
- Copilot Studio builder experience (Phase 3, separate team)
2
Back-of-envelope estimation
Scale numbers
Metric
Value
M365 MAU
400M
Copilot licensed users
30M
Copilot DAU
10M (~33% of licensed)
Daily Copilot invocations
150M (~15 per DAU)
Peak RPS
5K (3× average, business hours)
Peak concurrent sessions
50K
Token estimation
Token component
Avg tokens
System prompt (metaprompt + app instructions)
1,000
Retrieved context (Semantic Index + active doc)
5,000
User query
500
Conversation history
1,500
Total input per request
8,000
Output per request
2,000
Daily tokens
150M × 10K ≈ 1.5T tokens/day
Cost estimation
Resource
Monthly cost
Azure OpenAI inference (GPT-4, 1.5T tok/day)
~$20M
Semantic Index (vector search infra)
~$5M
Copilot Orchestrator (compute + networking)
~$1.5M
Responsible AI filters
~$1M
Microsoft Graph API (incremental)
~$500K
Storage (conversations, audit, caches)
~$200K
Total
~$28M/month
★ STAFF SIGNAL
LLM inference is ~70% of total cost. Routing 60% of requests to GPT-4o-mini instead of GPT-4 saves ~$12M/month. At $30 × 30M = $900M/month revenue, $28M infra is ~3% of revenue — healthy margin. GPU capacity is the binding scaling constraint, not compute or storage.
3
API design
External API (client-facing)
Copilot endpoints
# Primary Copilot endpoint (SSE streaming)
POST /v1.0/copilot/chat/completions
Headers: Authorization: Bearer {Entra_ID_token}
Body: {
"app_context": {
"app": "word",
"active_document_id": "driveItem/abc123",
"selection": { "start": 0, "end": 500 }
},
"conversation_id": "conv-uuid-001", // null = new conversation
"user_message": "Summarize the key findings from Q3",
"grounding_sources": ["semantic_index", "active_document"],
"plugins_enabled": ["microsoft.graph", "salesforce.crm"]
}
Response: SSE stream of token chunks + citations + metadata
# Feedback (RLHF + quality tracking)
POST /v1.0/copilot/feedback
Body: { "turn_id", "rating": "thumbs_down", "reason": "inaccurate" }
# Plugin registration
POST /v1.0/copilot/plugins
Body: { "manifest_url", "category": "third_party", "declared_apis": [...] }
Inter-service contracts
gRPC / HTTPS service SLAs
SERVICE: CopilotOrchestrator PROTOCOL: gRPC (streaming)
LATENCY p99 < 5s TTFT AVAILABILITY 99.9%
RPC ProcessCopilotRequest(CopilotRequest) -> stream CopilotChunk
Auth: mTLS + Entra ID identity forwarded from APIM gateway
SERVICE: SemanticIndex PROTOCOL: gRPC
LATENCY p99 < 500ms AVAILABILITY 99.9%
RPC RetrieveContext(RetrievalRequest) -> RetrievalResponse
Security-trimmed to user permissions; hybrid vector + BM25 with RRF
SERVICE: AzureOpenAIInference PROTOCOL: HTTPS (SSE)
LATENCY p99 < 3s TTFT (GPT-4), < 1s (GPT-4o-mini) AVAIL 99.9%
POST /openai/deployments/{model}/chat/completions
Reserved Copilot-dedicated inference pools; SSE token-by-token
API governance
- Versioning — URI for Graph (/v1.0/), query param for Azure OpenAI (?api-version=)
- Rate limiting — per-user 30 req/min; per-tenant scaled by seat count (1000 seats = 5000 req/min)
- Backpressure — HTTP 429 with Retry-After; exponential backoff enforced by the client SDK
4
Data model
Core entities
entities
CopilotConversation
conversation_id PK, tenant_id (partition), user_id, app ENUM,
created_at, last_active_at, turn_count, status ENUM
CopilotTurn
turn_id PK, conversation_id FK, tenant_id (partition),
user_message, assistant_response, model_used,
input_tokens, output_tokens,
retrieved_sources JSON [{source_id, title, snippet, score}],
plugins_invoked JSON, safety_flags JSON, latency_ms
SemanticIndexEntry
entry_id PK, tenant_id (partition), user_id, source_type ENUM,
source_id, chunk_id, content_text, embedding_vector FLOAT[1536],
permissions_hash (ACL hash for security trimming), source_modified_at
PluginRegistration
plugin_id PK, tenant_id (null = global first-party), name,
manifest_url, category ENUM, declared_apis JSON,
oauth_config JSON (encrypted), status ENUM
CopilotAuditLog
log_id PK, tenant_id (partition), user_id, turn_id,
action ENUM, details JSON, data_residency_region, created_at
Data ownership boundaries
Data
Owner
Access for others
Conversations & turns
Orchestrator
Sync gRPC (by conversation_id)
Semantic index entries
Semantic Index
Sync gRPC (security-trimmed)
Plugin registrations
Plugin Platform
Sync gRPC (cached, invalidated on status change)
Documents, emails, calendar
Microsoft Graph
Sync REST (cached)
Audit logs
Compliance / Purview
Async (Service Bus), query via Purview
Profile & permissions
Entra ID / RBAC
Sync gRPC (cached, 5-min TTL)
Storage technology choices
Data
Technology
Rationale
Rejected
Conversations
Cosmos DB (part: tenant_id)
Global distribution, geo-replication, sub-10ms reads
Azure SQL (single-region)
Index vectors
Azure AI Search
Hybrid vector + keyword, integrated security trimming
Pinecone (EU residency)
Session cache
Azure Redis
Sub-ms ephemeral session state, TTL eviction
Cosmos DB (overkill)
Audit logs
Blob (append)
Cheap at PB scale, immutable append, tiering
Cosmos DB (10× cost)
Plugin manifests
Cosmos DB
Low volume, global reads, strong consistency on approval
Azure SQL
Data lifecycle
HOT (0-24h): active conversations in Redis + Cosmos DB
WARM (1-30d): conversations in Cosmos DB (autoscale RU), full indexing
COLD (30-365d): archived to Blob (cool tier); audit logs retained
PURGE (per-tenant): crypto-shredding for GDPR —
delete the key, all tenant data becomes unrecoverable
5
High-level architecture
Architecture diagram
[Azure Front Door]
(Global LB, WAF)
|
+---------------+---------------+
| |
[US-West Region] [EU-West Region]
| |
[API Management] [API Management]
(auth, rate limit, routing) (auth, rate limit, routing)
| |
[Copilot Orchestrator] [Copilot Orchestrator]
| | | | | | | |
[Semantic Model Plugin RAI [Semantic Model Plugin RAI
Index] Router Runtime Filter] Index] Router Runtime Filter]
| | | |
| [Azure OpenAI Service] | [Azure OpenAI Service]
| (GPT-4 / 4o / 4o-mini) | (GPT-4 / 4o / 4o-mini)
+----+ +----+
| |
[Async Pipeline: Service Bus] [Async Pipeline: Service Bus]
[Audit] [Analytics] [RLHF] [Audit] [Analytics] [RLHF]
Service boundaries with ownership
Service
Team
Store
Scaling
Consistency
SLA
API Management
Platform
—
Horizontal/region
—
p99 < 15ms
Orchestrator
Orchestrator
Cosmos + Redis
Horizontal (stateless)
Eventual (cache)
p99 < 5s
Semantic Index
Sem. Index
AI Search
Partitions per tier
Eventual (< 5min)
p99 < 500ms
Model Router
Orchestrator
—
Horizontal
—
p99 < 10ms
Azure OpenAI
AOAI Platform
GPU weights
Reserved pools
—
p99 < 3s
Plugin Runtime
Plugin Plat.
Cosmos (manifests)
Sandboxed containers
Strong (reads)
p99 < 5s
RAI Filter
Responsible AI
Classifiers
Horizontal
—
p99 < 200ms
Audit Logger
Compliance
Blob
Async consumers
Eventual
99.9% delivery
Sync vs async communication
Communication
Type
Rationale
Client → Orchestrator
Sync (SSE)
User waiting for response tokens
Orchestrator → Semantic Index
Sync (gRPC)
Context needed before prompt construction
Orchestrator → Azure OpenAI
Sync (HTTPS)
User waiting for generated tokens
Orchestrator → Plugin Runtime
Sync (5s timeout)
Plugin results needed in prompt
Orchestrator → RAI Filter
Sync (200ms)
Safety must clear before serving
Orchestrator → Audit Logger
Async (Service Bus)
Fire-and-forget, guaranteed delivery
Graph → Semantic Index
Async (notifications)
Doc changes trigger re-indexing
Failure-domain boundaries
FD1 LLM inference (Azure OpenAI)
Fallback chain GPT-4 -> GPT-4o -> GPT-4o-mini; else graceful message
FD2 Semantic Index
Degrade to non-RAG: use only active-document context; warn user
FD3 Plugin Runtime
Skip plugin, text-only response; inform user data was unavailable
FD4 Responsible AI Filter
FAIL-CLOSED — block the response. Unfiltered output is NEVER served.
The only domain where unavailability beats degradation.
FD5 Async pipeline (audit, analytics)
Zero user impact; Service Bus durable retry; backfill on recovery
6
Deep dive
WHERE STAFF IS WONDecision 1 — RAG vs fine-tuning vs long context
Approach
Per-tenant cost
Freshness
Privacy
Scalability
RAG via Semantic Index
Low (shared infra)
Real-time (< 5min)
Security-trimmed at query
Scales to 30M+ tenants
Fine-tune per tenant
Very high ($5K+/update)
Stale (days/weeks)
Tenant data in weights
Impossible at 30M
Long context only
Moderate (tokens)
Real-time
Ephemeral
Limited by 128K window
Hybrid: RAG + active doc
Low–moderate
Real-time
Trimmed + ephemeral
Best of both
★ STAFF SIGNAL
Recommendation — hybrid RAG + long context. Semantic Index for broad retrieval across all user data; inject the active document directly into the long-context window. Grounding across the user’s entire M365 corpus while preserving full fidelity of the doc they’re actively editing.
RAG pipeline (6 steps)
1 Query understanding (<50ms): intent, target entities, query rewrite
2 Context assembly (parallel, <400ms):
a) active document inline (truncate to 4K, keep selection)
b) Semantic Index: embed -> hybrid vector+BM25 -> RRF merge
-> security-trim to permissions -> top-10 chunks
c) Graph metadata (<200ms): resolve entities, recent activity
3 Prompt construction (<50ms): metaprompt + chunks + active doc
+ history + query; citation markers per chunk
4 LLM inference (streaming, <3s TTFT): route to selected model
5 Post-processing (<200ms/chunk): RAI post-filter, grounding check,
citation insertion
6 Response delivery: stream tokens as they clear post-processing
Decision 2 — multi-model routing
Model
In /1K
Out /1K
TTFT
Quality
Best for
GPT-4 Turbo
$0.01
$0.03
~1.5s
95
Complex analysis, long drafts
GPT-4o
$0.005
$0.015
~800ms
92
Summarization, multi-step reasoning
GPT-4o-mini
$0.00015
$0.0006
~300ms
82
Email reply, formula hint, simple Q&A
routing algorithm
signals: app, query_complexity (tokens, entities, depth),
context_volume, past_feedback, tenant_tier
Phase 2 target mix:
email_reply, formula_hint, quick_qa -> GPT-4o-mini (60%)
summarization, recap, rewrite -> GPT-4o (25%)
complex_analysis, long_draft, multi_hop -> GPT-4 Turbo (15%)
fallback: 429/timeout -> next model up in quality
cost: GPT-4 only ~$20M/mo -> routed mix ~$8M/mo (~$12M saved)
Decision 3 — tenant isolation & data residency
Model
Isolation
Cost
Compliance
Use case
Shared logical
tenant_id filter at every layer
Low
SOC 2, GDPR
Standard commercial
Dedicated
Separate pool + index
Very high
FedRAMP High, ITAR
Government (GCC High)
Tiered
Logical for standard, physical for regulated
Moderate
Full spectrum
Recommended
★ STAFF SIGNAL
Recommendation — tiered isolation. Standard commercial tenants share infrastructure with strict logical isolation; regulated tenants (GCC High, EU Data Boundary, HIPAA) get physically separate inference pools and index partitions.
7-step isolation protocol
1 Extract immutable tenant_id from Entra ID token at APIM gateway
2 Scope Semantic Index query to tenant_id partition + security trim
3 tenant_id is NEVER in prompt text sent to the LLM
4 Route to tenant-assigned pool (shared vs sovereign for GCC High)
5 No persistent KV cache across requests (no attention leakage)
6 Audit log written to tenant residency region (EU stays EU)
7 Crypto-shredding on deletion: destroy the tenant key
Decision 4 — plugin architecture & LLM tool use
function-calling flow (5 steps)
1 LLM emits tool_calls, e.g. {"name":"salesforce.get_customer",
"arguments":{"id":"C-12345"}}
2 Orchestrator validates: plugin approved for tenant? user permitted?
API in declared_apis manifest?
3 Acquire Entra ID on-behalf-of token, min scope, cached 5min
4 Execute in sandbox, 5s hard timeout:
first-party -> direct Graph call (trusted, no sandbox)
third-party -> gVisor container, declared endpoints only,
1 vCPU / 512MB, output sanitized
5 Inject tool_result into context; LLM writes final cited response
security model
First-party (Graph, SharePoint, Dynamics):
direct Graph API, no sandbox, full M365 permission model
Third-party (Salesforce, ServiceNow, SAP):
gVisor sandbox, only declared APIs reachable, output sanitized,
rate limit 100/hr/plugin/user, admin approval, manifest reviewed
for prompt-injection patterns
Decision 5 — safety architecture (three layers)
L1 Pre-filter (<100ms, FAIL-OPEN):
harmful-query classifier, PII redaction, indirect prompt-injection
scan of retrieved docs ("ignore previous instructions ...")
L2 System-prompt hardening (at construction):
metaprompt constraints, retrieved docs wrapped in role-marked XML,
app-specific limits (Excel suggests formulas, never mutates data)
L3 Post-filter (<200ms/chunk, FAIL-CLOSED):
content safety, grounding verification, citation accuracy,
cross-user PII leak detection — block on flag, never serve unfiltered
★ STAFF SIGNAL
Safety fails closed. The RAI post-filter is the one failure domain where unavailability beats degradation — if it’s down or flags a chunk, the response is blocked. Unfiltered output is never served.
7
Multi-team rollout strategy
Phase 1 — Foundation (weeks 1–10)
- Word + Outlook only, US-West single region, GPT-4 only (no routing)
- Internal dogfood (weeks 1–6) → 1% of licensed tenants (weeks 7–10)
- Success: TTFT p99 < 5s, grounding > 85%, zero cross-tenant leaks
- Manual review of 1% of responses for quality and safety
Phase 2 — Scale (weeks 11–24)
- All M365 apps; 5 regions; multi-model routing enabled
- First-party plugins (Graph, SharePoint, Dynamics)
- Progressive rollout 1% → 5% → 25% → 100% of licensed tenants
- Shadow routing: send queries to both GPT-4 and the routed model, compare quality before switching live traffic — GPT-4 stays source of truth until parity
Phase 3 — Platform (weeks 25–40)
- Third-party plugin marketplace (Salesforce, ServiceNow, SAP)
- Copilot agents (multi-step workflows), Copilot Studio for enterprises
- GCC High sovereign fleet; EU Data Boundary; full Purview audit trail
★ STAFF SIGNAL
Migration note — Copilot is net-new (no legacy to migrate, no strangler-fig). The real migration risk is model routing: shadow-route to both GPT-4 and the cheaper model and confirm quality parity before shifting live traffic. Never degrade quality to save cost.
Rollback plan
Stage
Rollback action
Data impact
Phase 1 (flag)
Feature flag off — Copilot UI hidden
None, conversations retained
Phase 2 routing
Revert routing — all traffic back to GPT-4
None, higher cost
Phase 2 region
Drain unhealthy region via Front Door
None, higher latency
Phase 3 plugins
Admin kill switch per plugin / all third-party
None, no plugin data
8
Bottlenecks, observability & evolution
Bottleneck analysis
Component
Bottleneck
Mitigation
Azure OpenAI GPU
Finite GPU, 429s at peak
Reserved Copilot pools, multi-model routing, request batching
Semantic Index
Heavy users (100K+ docs) spike latency
Tiered index: hot (recent 1K, in-mem), warm (all, disk); query hot first
Orchestrator fan-out
Sequential Index→Prompt→LLM→Safety adds up
Parallelize Index + Graph + pre-filter; start LLM while index returns
128K context limit
Large doc + history exceeds window
Hierarchical summarization, iterative retrieval
Plugin latency
Third-party APIs slow/unreliable
5s timeout, response cache (5min for idempotent GETs), proceed on timeout
Inference cost
$20M/mo at GPT-4-only
Routing (60% savings), prompt caching (15%), response caching (10%)
Observability (RED method)
Service
Rate
Errors
Duration
Orchestrator
req/s, tokens/s streamed
5xx, timeout, safety-block rate
TTFT p50/p99, total time
Semantic Index
queries/s, docs scanned/s
zero-result, permission-denial
query latency p50/p99
Azure OpenAI
tokens/s, req/s
429, timeout, content-filter rate
TTFT p50/p99, TPS
Plugin Runtime
invocations/s per plugin
error, timeout, sandbox OOM
exec latency p50/p99
RAI Filter
chunks filtered/s
false pos / false neg (sampled)
filter latency p50/p99
Critical alerts
- TTFT p99 > 5s (5 min sustained) — model / index / orchestrator bottleneck
- Grounding accuracy < 85% (hourly) — index degradation or retrieval regression
- Cross-tenant data leak — SEV-0: global Copilot disable, all-hands, customer notification
- Azure OpenAI 429 rate > 5% — capacity exhaustion, activate overflow pools
- Safety block rate > 10% — prompt-injection campaign or RAI false-positive spike
Distributed tracing
Application Insights span tree
[click] -> [Front Door] -> [APIM: auth + rate limit]
-> [Orchestrator]
PARALLEL: [Index 400ms] [Graph 200ms] [RAI pre-filter 100ms]
-> [Model Router 10ms] -> [Azure OpenAI 1-3s TTFT]
PER CHUNK: [RAI post-filter 200ms] -> [client token]
-> ASYNC: [audit] [analytics] [RLHF]
span dims: tenant_id, user_id, conversation_id, turn_id, app,
model_used, tokens, grounding_score, safety_result
3-year evolution
- Year 1 — RAG Copilot across all apps; routing 60/25/15; 50+ plugins; GCC High; < 15min MTTR
- Year 2 — multi-step agents; Copilot Studio; industry fine-tunes; proactive Copilot; prompt caching
- Year 3 — Copilot SDK for ISVs; cross-app orchestration; on-device small models; multi-modal voice/image
Cost-optimization roadmap
Strategy
Timeline
Savings
Multi-model routing (60% to mini)
Month 3
40% inference
Prompt caching (common prefixes)
Month 4
15% tokens
Response caching (identical queries)
Month 6
10% inference
Tiered index storage (hot/warm)
Month 9
30% index
Reserved GPU (1-yr commit)
Month 6
25% inference
Batch mode (overnight reports)
Month 12
50% for batch
✓
Summary — key Staff-level insights
1. RAG via Semantic Index, not fine-tuning — the only architecture that scales to 30M+ tenants economically; per-tenant fine-tuning is financially and operationally impossible at this scale.
2. Multi-model routing is the single biggest cost lever — shifting 60% of traffic to GPT-4o-mini saves ~$12M/month while holding quality through task-appropriate model selection.
3. Tiered tenant isolation — logical for standard commercial, physical for regulated/government (GCC High) — balances cost efficiency against compliance.
4. Plugin security requires sandboxed execution — third-party plugins run in gVisor with declared-API-only egress, output sanitization, and prompt-injection defense.
5. Safety must fail closed — the RAI post-filter is the only failure domain where unavailability is preferable to serving unfiltered output.
6. A 3-year arc: RAG assistant → agents → platform — Year 1 nails the foundation, Year 2 adds agentic workflows, Year 3 opens Copilot as an extensible platform.
★
Rubric — Senior vs Staff
Dimension
Senior signal
Staff signal
Requirements
Lists functional + latency targets
Splits short- vs long-form TTFT; challenges the isolation model up front
Architecture
Clean orchestrator + RAG + inference flow
Justifies RAG over fine-tuning at 30M-tenant scale
Cost & scale
Estimates tokens and monthly spend
Names model routing as the $12M/mo lever; GPU as the binding constraint
Tenant isolation
Filters by tenant_id everywhere
Tiered logical vs physical; crypto-shredding; residency-scoped audit logs
Plugin security
Validates manifests, OAuth on-behalf-of
Sandboxes third-party in gVisor; declared-API egress; output sanitization
Safety
Pre/post content filters
Three layers; post-filter fails CLOSED — never serves unfiltered output
Staff ownership
Defines dashboards + alerts
Owns SEV-0 leak response, shadow-routing quality gates, 3-year platform arc
★ MORE WALKTHROUGHS
Want more breakdowns like this?
Join free early access for upcoming RAG, LLM eval, agents, and AI infrastructure walkthroughs.