Your database failed over. Is your position still real?

2026-07-05 • by Vahid Negahdari

This is the fourth post in a series on the hard parts of building a Change Data Capture engine, told through DeltaForge, a CDC engine I’m writing in Rust. The previous post was about the handoff from snapshot to stream. This one is about what happens when the database you are streaming from is quietly replaced by a different one.

A reconnect is not a reconnect

CDC engines reconnect all the time. Connections drop, brokers restart, a deploy cycles the pod. The normal thing to do is reconnect and carry on from the last checkpoint, and almost always that is correct.

Failover is the case where it is catastrophically wrong. The connection comes back, the query succeeds, everything looks healthy, but the server on the other end is not the same server. A replica got promoted. The primary you were reading from is gone, and the new one has its own history that may not include the position you were about to resume from.

The trap is that at the protocol level a failover looks exactly like a reconnect. If your engine treats it as one, it resumes from a checkpoint that means nothing on the new server, and depending on the database that either throws a confusing error or, much worse, silently skips everything the old primary had that the new one never received.

Step one: is this even the same server?

You cannot answer “is my position still valid” until you answer “am I still talking to the same database.” So on every reconnect, before anything else, DeltaForge asks the server for its stable identity: @@server_uuid on MySQL, system_identifier from pg_control_system() on Postgres. Not “is the connection up,” but “who are you, really.”

That identity is compared to the one stored durably from last time, and there are three answers:

The interesting detail is that Same still triggers a reachability check. Same server does not mean same history: someone can run RESET BINARY LOGS AND GTIDS and wipe all the GTID state without the UUID changing at all. Identity narrows the question; it does not answer it.

Step two: does the new primary still have my position?

This is the heart of it. The engine’s checkpoint came from the old primary. The only question that matters is whether the new primary has already executed every transaction in it. On MySQL that is a single function the server will answer for you:

// on the NEW primary: does it already contain every transaction we acked?
let q = format!("SELECT GTID_SUBSET('{saved_gtid}', @@global.gtid_executed)");
match conn.query_first::<i64, _>(&q).await? {
    Some(1) => PositionReachability::Reachable,             // safe to resume
    Some(0) => PositionReachability::Lost   { reason },     // halt, re-snapshot
    _       => PositionReachability::Unknown { reason },    // transient: warn, keep going
}

Three outcomes, and each one is a deliberate decision.

Reachable means the new primary is caught up past our checkpoint. Safe to resume. Lost means it is not: there are transactions we already acknowledged that the new primary never received, because it was a lagging replica when it got promoted. Those rows are gone at the database level, and no CDC engine can conjure them back. Unknown means the check itself failed transiently (a flaky health query), and this one is subtle: DeltaForge does not halt on uncertainty. Halting because a health query timed out would turn every network blip into an outage. It warns and continues, and re-checks on the next reconnect.

When the answer is Lost, the source stops immediately and /health returns 503 with position lost: <reason>. Re-snapshot required. That is the whole philosophy in one line: silently skipping data is worse than halting.

The demo is that decision. The engine’s checkpoint sits just behind the head of the old primary. Drag the new primary’s replica lag and fail over. While it is caught up, the resume is clean; push it past the checkpoint and the acked transactions in between are gone, and you can watch DeltaForge halt on it or, with guards off, skip it without a word.

The subtle trap: the fix that loses data on its own

Here is the part that took me longest to get right, and it is the reason failover has to be detected before the stream reopens, not during reconciliation.

The naive recovery is “just reconnect at the checkpoint position.” On Postgres that single act can destroy data before any check runs. Calling START_REPLICATION at your old LSN advances the slot’s confirmed_flush_lsn to max(your_lsn, slot_lsn). If the new primary’s slot sat behind your checkpoint, the changes it committed in that gap are discarded the instant you connect, permanently, even if you then realize your mistake and reconnect at the right place. The act of asking is what deletes the evidence.

MySQL is less destructive but still wrong: the new primary rejects your GTID set outright with “purged required binary logs.”

So DeltaForge detects the identity change before it ever opens the replication stream. It resolves the new primary’s actual current position (its binlog tail, or the slot’s real confirmed_flush_lsn) and starts there, while keeping the old checkpoint aside, untouched, purely as input to the reachability check. The position it streams from and the position it validates are two different values on purpose. Conflating them is how you delete data while trying to check whether you lost any.

When the schema moved too

A promoted replica can also have drifted. Maybe a migration reached A but not B before promotion, so B is missing a column that A had, and row events encoded against A’s schema will decode wrong against B’s.

So reconciliation also diffs the last-registered schema against the new primary’s live catalog and records what changed. The policy is yours:

source:
  config:
    on_schema_drift: halt   # default: adapt

adapt records the drift, reloads the schema, and continues, which is safe when the drift is additive. halt stops on any drift and waits for a human, which is the right default when your failover environments do not guarantee that DDL has synced to replicas before one gets promoted. Either way the drift is written down before anything proceeds, so you can see what moved.

What it does not pretend to do

Being honest about the boundaries is most of the value here.

DeltaForge does not switch the DSN for you. It detects a new server by comparing identities, not by watching cluster topology, so the connection string has to already point at the new primary by the time it reconnects. That is a job for a VIP, DNS failover, or a proxy. If the DSN still resolves to the dead primary, DeltaForge will loyally keep retrying a corpse.

And it cannot recover data that a lagging replica never received. It can detect the gap, name it, and refuse to paper over it, but the rows A committed and B never saw are gone. The honest response to that is a re-snapshot from B, which is exactly what the 503 tells you to do. For any of this to work at all, MySQL needs GTID mode on (file and position coordinates are meaningless across servers), and Postgres needs the replication slot to already exist on the standby, which is a job for a slot-aware HA tool like Patroni.

The series, in one sentence

Four posts, four hard problems: inferring the shape of data nobody declared, delivering it honestly across mismatched sinks, handing off from snapshot to stream without a silent gap, and now surviving a failover without silently skipping. If there is one thread running through all of them, it is this: in a CDC engine the failures that hurt are never the loud ones. They are the two subsystems that each did their job, meeting at a seam nobody guarded, producing a result that looks like success and is quietly wrong. The engineering is mostly finding those seams first.

The code is at github.com/vnvo/deltaforge; failover reconciliation lives in the sources crate (mysql_health.rs, postgres_health.rs) and the chaos suite has failover, pg_failover, binlog_purge, and slot_dropped scenarios that try to break exactly this. It is pre-release, and I would genuinely like people to try to break it.