Your snapshot succeeded, and you still lost data

2026-06-28 • by Vahid Negahdari

This is the third post in a series on the hard parts of building a Change Data Capture engine, told through DeltaForge, a CDC engine I’m writing in Rust. The second post was about delivering changes without losing or duplicating them. This one is about the step before any of that: getting the existing data in, and the quiet way that step corrupts itself.

Two jobs that have to line up perfectly

When you point CDC at a database that already has data, you have two jobs. First, copy everything that is already there, the initial snapshot. Second, stream every change from that moment forward. The whole thing only works if those two line up exactly: the stream has to start at the precise log position that matches the snapshot, with no gap and no overlap.

That sounds like bookkeeping. It is actually the hardest correctness problem in a CDC source, because the snapshot takes real time (minutes, sometimes hours, on a large table), the database keeps changing the entire time, and the transaction log you plan to resume from is being trimmed underneath you the whole while.

Get the handoff slightly wrong and you do not get an error. You get a database replica that is quietly missing a few minutes of history, forever, and no one notices until a number does not add up months later.

Getting the copy consistent (the easy hard part)

The first half has a known answer, and DeltaForge uses it. You do not lock the tables. You take a consistent read.

On MySQL, every snapshot worker opens its connection with START TRANSACTION WITH CONSISTENT SNAPSHOT. InnoDB gives all of them the same point-in-time view of the database, so they can read different tables in parallel and still see one coherent snapshot, with no FLUSH TABLES WITH READ LOCK and no global stall. On Postgres it is the same shape with different primitives: a coordinator connection exports a snapshot and every worker imports it into its own REPEATABLE READ transaction, so they all read the same frozen state.

The snapshot rows are emitted as Op::Read events (Debezium’s op: "r"), so consumers can tell backfilled rows apart from live changes. So far, so textbook.

The half that bites: capturing the position

The subtle part is the log position, the thing streaming resumes from. Capture it a moment too early and you miss changes; too late and you double-count. And on MySQL the ordering is not what you would guess.

DeltaForge captures the binlog position after all the worker transactions have opened, not before. That feels backwards until you remember what the consistent snapshot guarantees: every row those transactions can see was committed at or before the position you read once they are all open. So the position captured after the workers start is a safe lower bound. Streaming from it re-sees nothing the snapshot already has, and misses nothing it does not.

/// Returns a checkpoint captured after all worker transactions open.
/// InnoDB guarantees every visible row was committed at or before this
/// position, so streaming resumes with no gaps.
pub async fn run_snapshot(ctx: &SnapshotCtx<'_>, tables: &[(String, String)])
    -> Result<MySqlCheckpoint>

Postgres folds the two steps into one: the coordinator exports the snapshot and reads the current WAL LSN in a single round trip, and that LSN becomes the resume point. Either way, at the end of the snapshot you are holding a position that is exactly right.

Or rather, a position that was exactly right at the moment you captured it. Which brings us to the landmine.

The landmine: retention outruns the snapshot

A transaction log is not kept forever. MySQL purges binlog files older than binlog_expire_logs_seconds. Postgres holds WAL for a replication slot, but a slot can be invalidated. The retention window is finite, and it slides forward as the database runs.

Now stack the timelines. You capture position P at the start of the snapshot. The snapshot reads for, say, forty minutes. Your binlog retention is thirty. Forty minutes later the snapshot finishes cleanly, every row emitted, no errors. But position P aged out of the retention window ten minutes ago. It is gone. When streaming tries to resume from P, the file it needs has been purged.

A naive engine does not notice, because nothing in the snapshot itself failed. It writes “snapshot complete,” starts CDC from P, the broker says “that position is too old, here is the oldest I have,” and the engine happily starts streaming from there. Everything between the purge and now is simply absent. The snapshot succeeded. You still lost data.

The demo makes the race concrete. Set the snapshot longer than retention and run it. Toggle the guards off to watch the naive version report success while losing data, and on to see what DeltaForge does instead.

The whole problem is that the failure lives in the gap between two subsystems that each think they did their job. The snapshot succeeded. The stream started. Neither one is wrong on its own, and that is exactly why it stays silent.

Three guards that make the gap loud

DeltaForge cannot make a thirty-minute retention hold a forty-minute snapshot. Nothing can. What it can do is refuse to ever turn that situation into a silent success. It wraps the snapshot in three checks.

Before it reads a single row, a preflight. It hard-fails on the misconfigurations that make CDC impossible at all (log_bin off, binlog_format not ROW). Then it estimates the snapshot duration from the table sizes and max_parallel_tables, and compares that estimate to your retention window. At fifty percent of retention it warns; at eighty percent it flags HIGH RISK. You find out you are cutting it close before the forty minutes, not after.

While it runs, a background guard. A task polls SHOW BINARY LOGS every thirty seconds and checks that the file holding your captured position is still there. The moment it is purged, the guard aborts the snapshot rather than let it finish into a lie:

loop {
    interval.tick().await;                       // every 30s
    let available = show_binary_logs(&dsn).await;
    if !binlog_file_still_present(&available, &captured_file) {
        *abort_reason.lock().unwrap() = Some(format!(
            "binlog file '{captured_file}' purged during snapshot. \
             Increase binlog_expire_logs_seconds or reduce max_parallel_tables."
        ));
        cancel.cancel();   // stop now, do not hand off a dead position
        return;
    }
}

After every table completes, a final synchronous check before it writes the “done” flag. And that flag is the real point. Progress is tracked in a small record:

pub struct MysqlSnapshotProgress {
    start_position: String,   // captured before any rows were read
    done_tables: Vec<String>, // "db.table" values already complete
    finished: bool,           // true ONLY when the position is confirmed valid
}

finished = true does not mean “the rows were emitted.” It means “the position is confirmed reachable for CDC resume.” A naive engine treats emitting the last row as success. DeltaForge treats a verified handoff as success. Those are different events, and conflating them is the bug.

A bonus from tracking it properly: resumable snapshots

That same MysqlSnapshotProgress record buys something else. Because it lists the tables already done, a snapshot that gets interrupted, a crash, a deploy, a killed pod, resumes at table granularity. On restart it skips the completed tables and picks up the rest, instead of throwing away six hours of backfill and starting the whole thing over. The captured start_position is preserved across the interruption too, so the eventual handoff still points at the right place.

Configuration stays boring, which is the goal:

source:
  type: mysql            # or postgres
  config:
    id: orders
    snapshot:
      mode: initial          # initial | always | never (default: never)
      max_parallel_tables: 8
      chunk_size: 10000

If a run warns you about retention, the fix is not in DeltaForge, it is in the setup: raise binlog_expire_logs_seconds to comfortably cover the estimated snapshot, or point the snapshot at a read replica so the load and the retention pressure are somewhere other than your primary.

The honest version

The guards do not make an impossible configuration work. A snapshot that takes longer than your log retention is going to fail, and it should. What DeltaForge changes is how it fails. Instead of a clean “snapshot complete” followed by a replica that is silently short a few minutes of history, you get a warning before you start, an abort the moment the position is actually lost, and a finished flag that only ever means what it says.

That is the recurring theme of building a serious CDC engine. The dangerous failures are almost never the loud ones. They are the two subsystems that each did their job, meeting at a seam nobody was watching. The engineering is mostly about finding those seams and putting a guard on them before they find you at 3am.

Next in the series

The snapshot assumes the database it is reading from stays the same database. The next post is about what happens when it does not: a failover swaps the primary out from under you mid-stream, and the engine has to decide whether the position it was tracking still means anything on the new server, or whether continuing would quietly skip everything the old primary had that the new one does not.

The code is at github.com/vnvo/deltaforge; the snapshot logic lives in the sources crate (mysql_snapshot.rs, mysql_health.rs, and the Postgres equivalents). It is pre-release, and the handoff path is the one I would most want a skeptical DBA to review.