System Design & Architecture Interview Question 6: Async Replication Failover Loss
System Design & Architecture Interview Questions · Test 1 · question 6 of 10
A ledger service runs a leader database with one follower in another datacenter. Replication is asynchronous. The leader's host crashed, and automated failover promoted the follower to leader within 30 seconds. Afterwards, customers report that several transfers they had received a "success" response for are missing from the new leader. No application errors were logged.
What is the most likely explanation?
Show answer and explanation
Answer:
With asynchronous replication, the leader commits a write locally, acknowledges it to the client, and ships it to the follower some time later. Anything in that replication-lag window exists only on the leader. If the leader dies before shipping it, promoting the follower loses writes that were already acknowledged. The size of that window is the system's recovery point. The fix is synchronous (or semi-synchronous) replication, where the leader acknowledges only after at least one follower has the write. The trade-off is added write latency, and writes may stall if no follower is reachable.
Option A is unlikely. A read-only follower rejects writes with an error instead of silently accepting and discarding them, and the clients received success responses.
Option B describes a mechanism that does not exist. Deduplication keeps one copy of a repeated request; it does not delete the original and leave nothing behind.
Option C would show stale reads that fix themselves as the cache refreshes. The transfers are missing from the new leader's data itself, which points to lost writes, not caching.
Reference: Architecture strategies for using availability zones and regions - Azure Well-Architected Framework