← Back to all questions
StaffInfrastructureBuild Systems
Design a Build System for a Billion-Line Monorepo
A Staff-level walkthrough of a Blaze/Bazel-style build system that turns a 30-minute local build of a billion-line monorepo into a two-minute one — built on a content-addressable remote cache, hermetic distributed execution, and dependency-graph test selection. The recurring tension is hit-rate and parallelism versus correctness, where under-selecting tests can silently ship regressions. Follow it top to bottom, or jump to any step.
100% free · No login required
WHAT THIS QUESTION TESTS
·Cache keys hash inputs, deps, and toolchain — never timestamps
·10K–100K independent actions fan out across the worker fleet
·Workers run read-only, networkless, and resource-capped for determinism
·Only tests transitively depending on changed files get run
★ STAFF-LEVEL SIGNALS
★Treats >95% cache hit rate as the load-bearing number, then designs for it
★Names under-selection as the real risk and backstops it with nightly full builds
★Justifies ~$18M/month against engineer-hours that slow builds would waste
★Decides local vs. remote builds by change size and cache locality
0
Scope & ambiguity
“Google’s monorepo has 1B+ lines of code with 50K+ engineers making 60K+ commits/day. The key challenge is making builds fast and reproducible at this scale. I’ll focus on remote caching, distributed execution, and test selection — the three pillars that make it possible to build from a 1B-line repo in minutes instead of hours.”
Teams: Build Infrastructure Team (build engine, caching), Execution Team (distributed build workers), Developer Experience Team (IDE integration, local development), CI/CD Team (continuous builds, presubmit checks).
1
Requirements
Requirement
Target
Incremental build time
< 2 minutes for typical change (touching <50 files)
Full build time
Not applicable (nobody builds everything; dependency graph determines scope)
Build reproducibility
100% (same inputs → same outputs, always)
Remote cache hit rate
> 95% (most build actions already computed by someone else)
Test selection accuracy
Only run tests affected by the change (< 5% false negatives)
Scale
50K engineers, 60K commits/day, 1B+ lines of code
2
Back-of-envelope estimation
Metric
Value
Total build actions per day
~500M (compile, link, test actions across all builds)
Cache entries
~10B (accumulated over time, with TTL)
Cache size
~500TB (compiled artifacts, test results)
Distributed build workers
~50K machines
Build requests per second
~50K (peak)
Cost
Resource
Monthly
Build worker fleet (50K machines)
~$15M
Cache storage (500TB SSD)
~$2.5M
Build orchestration servers
~$500K
Total
~$18M/month
Business justification: If builds took 10 minutes instead of 2, 50K engineers × 10 builds/day × 8 min wasted = 66K engineer-hours/day. At $100/hr, that’s $6.6M/day. The build system pays for itself many times over.
5
Architecture
[Developer's Change]
|
[Build Client (local)]
- Parse BUILD files
- Construct action graph (DAG of compile/link/test actions)
- Check remote cache for each action
|
[Remote Cache Service]
- Content-addressable store
- Key = hash(action inputs: source files, compiler, flags, deps)
- Value = action outputs (compiled object, test result)
- Hit rate: 95%+
| (cache miss)
[Distributed Execution Service]
- Execute action on remote worker
- Worker: hermetic environment (container with exact toolchain)
- Result stored back in cache
|
[Test Selection Service]
- Given changed files, compute affected test targets
- Only run tests whose transitive dependencies include changed files
|
[CI/CD Pipeline]
- Presubmit: run affected tests before merge
- Postsubmit: run broader test suite after merge
6
Deep dives
WHERE STAFF IS WONContent-Addressable Remote Cache
Build reproducibility requires: same inputs → same outputs.
Cache key computation:
action_key = hash(
source_file_hashes, // content hash of all input files
compiler_hash, // exact compiler binary hash
compiler_flags, // -O2, -std=c++17, etc.
dependency_output_hashes, // output hashes of all dependency actions
environment_variables, // relevant env vars
toolchain_hash // exact versions of all tools
)
If action_key exists in cache → download cached output (skip execution)
If not → execute action on remote worker → store result in cache
Why content-addressable (not timestamp-based):
- Timestamps are non-reproducible (same code at different times → different timestamps)
- Content-addressable means: if you revert a change and re-change it, cache still hits
- Enables sharing cache across ALL developers (your compile = my compile for same code)
Distributed Execution
When cache misses, execute on remote workers:
1. Build client sends action to scheduler
2. Scheduler assigns to worker based on:
- Resource requirements (CPU, memory, GPU)
- Locality (prefer worker that already has input files cached)
- Priority (presubmit > postsubmit > personal builds)
3. Worker executes in hermetic container:
- Read-only input filesystem (mounted from CAS)
- Write to output directory only
- No network access (reproducibility)
- Strict resource limits (timeout, memory)
4. Worker returns outputs → stored in cache → sent to build client
Parallelism:
- A typical build has 10K-100K actions
- Many are independent (parallelizable)
- With 50K workers: build parallelized across hundreds of machines
- A local 30-minute build → 2 minutes with distributed execution
Test Selection (Impact Analysis)
Problem: 1B lines of code has millions of test targets.
Running all tests for every change would take days.
Solution: Dependency graph analysis
1. Build system maintains a global dependency graph:
source file → build target → dependent targets → test targets
2. For a change touching files {f1, f2, f3}:
a. Find all build targets that directly include f1, f2, f3
b. Find all targets that transitively depend on those targets
c. Filter to test targets only
d. Run only those tests
3. Example:
Changed: //search/ranking/scorer.cc
Direct target: //search/ranking:scorer_lib
Dependent targets: //search/ranking:ranker, //search/serving:pipeline
Test targets: //search/ranking:scorer_test, //search/ranking:ranker_test,
//search/serving:pipeline_integration_test
Out of 5M total tests → only run 3 tests
Accuracy:
- Over-selection (false positives): acceptable, just slower
- Under-selection (false negatives): dangerous, could miss regressions
- Solution: periodic full-build verification (nightly) catches any gaps
Build vs Cloud Build Tradeoffs (Developer Laptop)
| Scenario | Where to Build | Why |
|----------|---------------|-----|
| Quick iteration (1-2 files) | Local + remote cache | Cache hit likely, no upload needed |
| Medium change (50 files) | Remote execution | Parallelism saves time |
| Large refactor (1000+ files) | Remote execution | Local machine can't handle it |
| Offline development | Local only | No remote access |
Developer laptop integration:
- Build client runs locally
- Fetches from remote cache for all dependencies
- Only compiles changed files locally OR sends to remote
- IDE integration: autocomplete/navigation uses cached build artifacts
7-8
Rollout & evolution
Rollout: Migrate teams from existing build system incrementally. Run both systems in parallel, compare build times and correctness. Language by language (C++ first, then Java, Python, Go).
Evolution: Year 1: Remote caching + distributed execution, 95% cache hit rate. Year 2: ML-based test selection (learn from historical test results which tests are most likely to fail), build performance analytics (identify slow targets, suggest optimizations). Year 3: Self-service build rules (teams define custom build rules without infra team), build cost attribution per team, integration with code review (show build/test results inline).
★
Rubric — Senior vs Staff
Dimension
Senior signal
Staff signal
Remote caching
Caches outputs so repeat work skips recompilation.
Content-addressable keys over hashed inputs and toolchain; explains cross-developer sharing and reproducibility.
Distributed execution
Offloads build actions to a remote worker pool.
Schedules by locality and priority; mandates hermetic, networkless workers for deterministic outputs.
Test selection
Runs tests near the changed files.
Walks the transitive dependency graph and treats false negatives as the danger, verified by periodic full builds.
Scale reasoning
Notes nobody builds the whole repo.
Quantifies ~500M actions/day and a 95% hit-rate target, and ties spend to engineer productivity.
Rollout
Moves teams onto the new system.
Runs old and new in parallel per language, comparing speed and correctness before cutover.
★ MORE WALKTHROUGHS
Want more breakdowns like this?
Join free early access for upcoming RAG, LLM eval, agents, and AI infrastructure walkthroughs.