Replication means keeping copies of the same data on multiple machines. These are my Designing Data-Intensive Applications Chapter 5 notes — in my own words, aimed at interviews.
The whole chapter in one line
Replication copies data across machines for availability, read scaling, and lower latency. Every hard problem in the chapter comes from one question: what happens when the copies disagree?
1 · Why replicate at all
Replication means keeping copies of the same data on multiple nodes. Three reasons, worth naming explicitly in an interview:
- High availability — if one node dies, another still serves the data.
- Read scaling — spread read traffic across many copies (one machine can't serve everything).
- Lower latency — put a copy geographically near users.
The catch: the moment you have more than one copy, you must keep them in sync — and syncing across a network that can delay or fail is where all the difficulty lives.
2 · Leader–follower (single-leader)
The most common scheme, used by PostgreSQL, MySQL, MongoDB, and many others.
- One node is the leader (primary). All writes go to the leader.
- The leader streams its changes (a replication log) to followers (replicas).
- Reads can be served by the leader or any follower — this is how you scale reads.
Why writes must funnel through one leader
If any node could accept writes, two nodes could change the same key at the same time with no agreed order → conflicts. A single leader gives every write one canonical order, which sidesteps conflicts entirely. That single benefit is the reason single-leader is the default.
Failover
If the leader dies, a follower must be promoted to new leader (automatically or manually). Failover is risky: choosing a stale follower can lose writes, and a network glitch can produce two nodes both thinking they're leader (split brain).
3 · Synchronous vs asynchronous replication
When the leader gets a write, does it wait for followers before telling the client "done"?
| Synchronous | Asynchronous | |
|---|---|---|
| Leader waits for follower? | Yes, before acking | No, acks immediately |
| Durability if leader dies | Strong — follower has the write | Weak — unreplicated writes can be lost |
| Write latency | Higher (waits on network) | Lower (fire and forget) |
| If follower is slow/down | Write blocks | Unaffected |
Semi-synchronous — the practical middle ground
Make one follower synchronous and the rest asynchronous. You get a guaranteed up-to-date copy for durability without blocking on every replica. Most real systems land here. Fully synchronous to all followers is impractical — one slow node stalls all writes.
4 · Replication lag & its consistency problems
With asynchronous replication, followers are slightly behind the leader — replication lag (usually milliseconds, sometimes more). A read from a lagging follower can return stale data. This produces three classic anomalies, each with a named guarantee that fixes it. This section is the highest-value part of the chapter for interviews.
4.1 · Read-after-write consistency
The anomaly
You update your profile, the write hits the leader, then your next read hits a lagging follower that hasn't received it yet → you don't see your own edit. Very confusing to users.
Guarantee: a user always sees their own writes. Fixes: read things the user may have modified from the leader; or for a short window after a write, route that user's reads to the leader.
4.2 · Monotonic reads
The anomaly
You refresh twice. First read hits an up-to-date follower (you see a new comment); second read hits a more-lagged follower (the comment vanishes). Time appears to move backward.
Guarantee: you never see data older than data you already saw. Fix: pin each user to always read from the same replica (e.g. hash their user ID to a replica).
4.3 · Consistent prefix reads
The anomaly
Writes that are causally ordered (question, then answer) arrive at a replica out of order, so an observer sees the answer before the question. Causality is violated.
Guarantee: writes that are causally related are seen in the correct order. Fix: ensure causally-related writes go through the same partition, or track causal dependencies.
Walk it through — two users, two partitions. A conversation happens in this real order:
Time 1 User A: "Are you free tomorrow?" → stored on Partition 1
Time 2 User B: "Yes, let's meet at 5pm" → stored on Partition 2
B's reply exists only because A asked — question causes answer. Now User C opens the chat to read it, and the two partitions are replicating at different speeds:
Partition 2 (answer) replica CAUGHT UP → "Yes, let's meet at 5pm" visible
Partition 1 (question) replica LAGGING → question NOT visible yet
C assembles the read from both partitions and sees:
User B: "Yes, let's meet at 5pm"
User A: (nothing yet…)
The answer appears before the question — an effect before its cause. That's a consistent-prefix violation.
Why it needs BOTH sharding and lag
Every sharded + replicated database is a grid — each partition has its own replicas that lag independently. Order is preserved within a partition, but sharding removes any global order across partitions, and each partition's replicas lag on their own. Single-partition systems can't have this anomaly at all — it takes the grid (multiple partitions × their own replicas) to create it.
PARTITION 1 PARTITION 2
├─ leader ├─ leader
├─ replica 1a (lagging) ├─ replica 2a (caught up)
└─ replica 1b └─ replica 2b
If you remember one thing from Chapter 5
The three guarantees by name: read-after-write (see your own writes), monotonic reads (time never goes backward), consistent prefix (causal order preserved). Naming these precisely is a strong signal in any distributed-systems interview.
And notice the shared lever behind all three fixes — routing: send the user's reads to a node that has their write (read-after-write); pin the user to one replica (monotonic); keep related writes on one partition (consistent prefix).
5 · Multi-leader replication
More than one node accepts writes; each leader is also a follower of the others. Used across multiple datacenters, or for offline-capable apps (each device is a "leader").
- Pro: writes can happen in multiple regions with low local latency; tolerates datacenter/network partitions better.
- Con: the same key can be written on two leaders concurrently → write conflicts you must resolve (see §8).
The core trade vs single-leader
Single-leader avoids conflicts by funneling writes through one node, at the cost of that node being a bottleneck / single write region. Multi-leader removes that bottleneck but reintroduces conflicts. You're trading a bottleneck for a conflict-resolution problem.
6 · Leaderless replication (Dynamo-style)
No leader at all. The client (or a coordinator) writes to several replicas directly and reads from several too. Used by Cassandra, DynamoDB, Riak.
- A write is sent to all N replicas; it succeeds once W of them acknowledge.
- A read queries multiple replicas and takes the newest value (using version numbers/timestamps).
- Lagging/failed replicas catch up via read repair (fix stale replicas during reads) and anti-entropy (a background sync process).
7 · Quorums: the W + R > N rule
The one formula to memorize for leaderless systems:
Quorum condition
W + R > N
N = replicas per item · W = replicas that must ack a write · R = replicas queried on a read.
Why it works: if the write set (W nodes) and read set (R nodes) must overlap, then any read is guaranteed to touch at least one node that has the latest write. Overlap = you can't miss the newest value.
Concrete: N = 3
Set W = 2, R = 2 → 2 + 2 = 4 > 3. ✓ Every read set of 2 overlaps every write set of 2 on at least one node. Tuning: raise W for stronger write durability, raise R for fresher reads, but higher W/R means lower availability (more nodes must respond).
Quorums aren't perfect
Even with W + R > N, edge cases (concurrent writes, a write that partially succeeds, sloppy quorums during partitions) can still return stale or conflicting data. Quorums give strong tunable consistency, not absolute guarantees.
8 · Conflict resolution
Multi-leader and leaderless systems must handle two writes to the same key that happened concurrently.
What "concurrent" actually means
Two writes are concurrent when neither one knew about the other when it happened — not necessarily the same millisecond. Single-leader can't produce this: every write passes through one node in a definite order, so write #2 always "knows about" write #1. Concurrency needs writes to land in multiple places that sync only after a delay — so each write is made in ignorance of the other.
How sync resolves a disagreement — two steps. Say two nodes end up disagreeing on one key:
Node 1: phone = "111"
Node 2: phone = "222"
Step 1 — Exchange: nodes send each other their recent writes. Now both nodes hold both values. Exchange alone doesn't fix anything — it just spreads the values. Step 2 — Resolve: apply a rule so all nodes converge on one final value:
if version info shows one write came AFTER the other:
→ clean overwrite, keep the later one (safe)
if the two writes are truly concurrent (neither saw the other):
→ LWW: keep latest timestamp (drops the other) ⚠ data loss
→ version vector: flag as a real conflict
→ siblings / CRDT: keep both / auto-merge (no loss)
The four resolution strategies:
- Last-write-wins (LWW): attach a timestamp, keep the latest. Simple but silently discards the other write — data loss, and clock skew can pick the wrong winner. (Cassandra's default.)
- Version vectors: track which writes are causally after which, so the system can tell a true conflict from a simple overwrite instead of blindly picking.
- Application-level merge (siblings): keep both versions and let the app or user resolve (e.g. merge two shopping carts by union). Refuses to lose data. (Riak.)
- CRDTs: data structures with a built-in merge rule that always converges regardless of order (counters, sets, carts) — no loss, no manual resolution.
Why more than one strategy exists
"Keep the latest" (LWW) is fine for a phone number — you probably do want the most recent. It's dangerous for a shopping cart, where silently dropping the other write loses items a user added. The right strategy depends on whether losing a concurrent write is acceptable.
9 · Interview cheat-sheet
| Model | Who writes | Conflicts? | Used by |
|---|---|---|---|
| Single-leader | One leader only | None (one write order) | PostgreSQL, MySQL, MongoDB |
| Multi-leader | Several leaders | Yes — must resolve | Multi-datacenter setups, CouchDB |
| Leaderless | Any replica (quorum) | Yes — must resolve | Cassandra, DynamoDB, Riak |
The soundbite
"Single-leader gives you one write order and no conflicts, but the leader is a bottleneck and a failover risk. Multi-leader and leaderless remove that bottleneck but reintroduce write conflicts you must resolve. Leaderless uses quorums — W + R > N — to tune the consistency/availability trade-off."
10 · Things to remember for interviews
Tiered by how often it actually comes up. If you're short on time, Tier 1 alone covers ~90% of what's tested.
Tier 1 — know cold
- The three models + when to use each: single-leader (one write order, no conflicts, bottleneck), multi-leader (multi-region writes, conflicts), leaderless (quorum, high availability, conflicts).
- The three lag guarantees by name: read-after-write, monotonic reads, consistent prefix. This is the single highest-value item.
- Quorum formula W + R > N — overlap guarantees every read sees the latest write. Be able to plug in N=3, W=2, R=2.
- Sync vs async — durability vs latency; semi-synchronous (one sync follower) is the practical middle ground.
Tier 2 — should know (follow-ups)
- Replication lag — async followers trail the leader → stale reads.
- The fix pattern is routing — all three lag guarantees are solved by controlling which replica/partition a related read or write goes to.
- Conflict resolution — LWW (simple, loses data), version vectors, siblings, CRDTs; and what "concurrent" means (neither write saw the other).
- Failover risks — split brain (two leaders), losing unreplicated async writes.
Tier 3 — bonus / senior signal
- Read repair & anti-entropy (how leaderless replicas catch up).
- Sloppy quorums & hinted handoff.
- Why quorums aren't a perfect consistency guarantee.
Three axes people wrongly collapse
Keep these independent — interviewers probe exactly this: (1) data model (SQL/NoSQL), (2) storage engine (B-tree/LSM), (3) replication mode (single/multi/leaderless). MongoDB = NoSQL + B-tree + single-leader proves all three are chosen separately. Single-leader is not a "SQL thing"; multi-leader is not a "NoSQL thing."
How it actually shows up
Not as a cold quiz — it's embedded in design questions. "You add read replicas to scale reads; what happens if a user reads right after writing?" → name read-after-write + the fix. "How does Cassandra stay available during a node failure?" → quorums, W + R > N.
11 · Interview drill
What is replication lag and why does it happen?
With asynchronous replication the leader acks a write without waiting for followers, so followers trail the leader by some delay (usually milliseconds). Reading from a lagging follower can return stale data. It's the price of not blocking writes on slow replicas.
A user edits their profile but doesn't see the change on reload. What's wrong and how do you fix it?
A read-after-write violation: the read hit a follower that hadn't received the write yet. Fix by reading data the user may have modified from the leader, or routing that user's reads to the leader for a short window after their write.
Explain the quorum condition W + R > N.
N replicas per item; a write needs W acks, a read queries R. If W + R > N, the read set and write set must overlap on at least one node, so every read sees at least one replica holding the latest write. Tune W up for durability, R up for fresher reads — at the cost of availability.
Single-leader vs multi-leader — when would you pick multi-leader?
When you need low-latency writes in multiple regions, offline-capable clients, or tolerance to a datacenter outage — situations where funneling all writes through one leader is too slow or too fragile. The cost is handling write conflicts, which single-leader avoids entirely.
What's the danger of last-write-wins conflict resolution?
It silently discards all but one concurrent write based on timestamp, so acknowledged writes can be lost — and clock skew can pick the "wrong" winner. It's simple and used by Cassandra, but unsafe when every write must be preserved; then prefer version vectors, sibling merges, or CRDTs.
What is split brain and why is it dangerous?
During failover or a network partition, two nodes both believe they're the leader and both accept writes, producing divergent data that's hard to reconcile. Systems guard against it with fencing tokens and consensus-based leader election (Chapter 9 territory).
The core takeaway: copies drift → name the guarantee that fixes each anomaly. Single-leader gives one write order and no conflicts but a bottleneck; multi-leader and leaderless remove the bottleneck but reintroduce conflicts; quorums (W + R > N) tune the trade-off. These are my own-words notes — if you spot an error or a cleaner framing, I'd like to hear it.