Would you notice a new field in your data pipeline?

2026-06-14 • by Vahid Negahdari

This is the first in a series on the genuinely hard parts of building a Change Data Capture engine, told through DeltaForge, a CDC engine I’m writing in Rust. DeltaForge tails a database’s transaction log (MySQL binlog, Postgres WAL) and streams every row change to sinks like Kafka, Redis, NATS, and S3. It is still pre-release, but the problems it has to solve are real, and this one is my favorite.

Most CDC stops at the column type

Every CDC tool tracks the source schema. It reads INFORMATION_SCHEMA on MySQL or pg_catalog on Postgres, learns that orders has an id BIGINT and a payload JSONB, and emits that structure alongside the data. When you run ALTER TABLE, it notices and adjusts. Call this schema tracking. It is table stakes, and DeltaForge does it too.

The trouble is that schema tracking tells you a column is jsonb, or payload TEXT, or event_data. It does not tell you what is inside. And in a lot of real systems, that is exactly where the interesting data lives: event envelopes, outbox bodies, denormalized documents, webhook captures, anything that got stuffed into a JSON column because the relational shape was inconvenient.

To the database, that column is an opaque blob. To everything that has to govern, secure, or use the data, it is the whole point:

• a platform team that owns the pipelines wants to see what is actually moving through them, not a column called payload
• a security or compliance team wants to know the day an email address, an access token, or a national id first appears in a stream, because that is a governance event whether or not anyone filed a ticket for it
• a data lake building Parquet columns wants the nested fields, typed
• a search index wants to know which keys exist
• an analytics warehouse wants to chart by an attribute buried three levels deep
• an AI agent answering questions about the data needs to know which fields exist and what they hold, so it can reference them directly instead of re-parsing a raw blob on every call

None of them can do anything useful with “it is a string.” They need the actual shape of the payload, and they need it tracked as it changes, without anyone maintaining a schema by hand.

That last part, tracking the shape as it changes, is where it gets interesting, and where it matters most to the platform and compliance cases. A genuinely new field is a signal worth acting on: it might be a feature, or it might be regulated data quietly entering a pipeline that was never reviewed for it. So the one thing you cannot afford is a sensor that cries “the schema changed” on every single event. That is exactly what naive payload inference does, and fixing it is the rest of this post.

Schema sensing: inferring the shape from the data

DeltaForge’s answer is a feature it calls schema sensing. Instead of reading the structure from database metadata, it infers the structure from the payloads themselves as they stream past. It samples events, walks the nested JSON, records which fields exist and what types they hold, and fingerprints the result so it can tell when the shape actually changes.

The distinction matters, so to be precise about it:

• Schema tracking reads structure from the catalog (DDL). Every CDC tool has it.
• Schema sensing infers structure from the data (payloads). Most CDC tools do not have it at all.

That second one is the differentiator. It also invites a fair objection, worth answering before going any further.

Why this belongs in the CDC engine

Discovering data shapes and flagging new fields sounds like a job for a data catalog or a governance scanner, not a CDC engine. The answer comes down to where you can see the data, and how early.

The CDC engine is the one component that sees every change, in order, at the source, before it fans out. A catalog scanning the warehouse sees the same opaque payload column, but it sees it later: after sampling, after transformation, and after the data has already been copied to every other sink. If an access token starts flowing into a stream at 2pm, the CDC layer can know at 2pm. The nightly catalog scan knows tomorrow, once the data is already in the lake, in Kafka, and in the search index. Catching it as it enters beats discovering it later, and the CDC stream is the only place you can catch it once, before it multiplies across every downstream system.

It is also the cheapest place to do it, and for DeltaForge it is not even a separate feature. The engine already deserializes every row and holds the payload structured in memory. More than that: to write a JSON payload into a typed Parquet column, or to register an Avro schema for a sink, it has to infer the payload’s shape anyway. Schema sensing is load-bearing for the sink encoding the engine already does. Exposing that inferred shape as an observability and governance signal is a short step, not a second system bolted on the side.

None of this replaces a catalog or a data-loss-prevention tool. The CDC engine is the sensor at the source, the earliest and most complete vantage point; the catalog is still the system of record. But if the question is whether a new field just entered a pipeline, the CDC stream is the first place, and the cheapest place, to answer it.

The failure mode: dynamic keys

Schema sensing sounds straightforward, right up until you point it at production JSON. Then it falls over almost immediately.

Here is a payload that looks innocent:

{
  "id": 1,
  "type": "page_view",
  "sessions": {
    "sess_abc123": { "user_id": 42, "started_at": 1700000000 },
    "sess_xyz789": { "user_id": 43, "started_at": 1700000001 }
  }
}

The sessions object is keyed by session id. The next event has different session ids. The event after that, different ones again. A naive structural fingerprint hashes the set of keys at every level, so every single event produces a brand new fingerprint. The sensor concludes the schema is evolving on every event.

The consequences are not subtle:

• the schema registry grows without bound, one “version” per event
• the structure cache hit rate sits at 0%, because no structure ever repeats
• downstream consumers get a “schema changed” signal constantly, which means the signal is worthless

And nothing actually changed. The shape is stable: an object with id, type, and a map of sessions. The keys of that map are data, not structure. The fix is to teach the sensor that difference.

You can watch the problem and the fix side by side. The stream below emits one event per tick, each with fresh random session ids. The left fingerprint counts every key. The right one collapses the detected dynamic map. Turn detection off on the right and watch it fall apart the same way:

The left side is what naive payload inference does: a new schema version on essentially every event, false evolution climbing toward 100%. The right side, with detection on, holds at one version no matter how many distinct session ids fly past.

Detecting dynamic keys, cheaply, online

The obvious fixes all fail:

• Treat every key as a field. That is the broken baseline above.
• Ignore keys you have not seen before. Now you cannot tell a genuine new field from a dynamic one, so you miss real schema changes.
• Scan every value and compare. Too slow to run on the hot path of a CDC stream doing tens of thousands of events per second.

What you actually want is to decide, per JSON path, whether its keys behave like fields (a small stable set that recurs) or like a map (an ever growing set of one off keys). That is a question about cardinality and frequency, and you have to answer it without storing every key you have ever seen, because storing them is the unbounded growth you are trying to avoid.

This is exactly what probabilistic sketches are for. DeltaForge leans on flowstats, a small streaming-algorithms crate I maintain, and tracks three things per path:

pub struct PathFieldStats {
    total_events: u64,
    total_field_observations: u64,
    cardinality: HyperLogLog,          // how many DISTINCT key names?
    heavy_hitters: SpaceSaving<String>, // which keys recur most?
    samples: ReservoirSampler<String>,  // a few real examples
}

HyperLogLog estimates the number of distinct key names seen at this path using a fixed block of memory, regardless of whether that is 5 distinct keys or 5 million. SpaceSaving tracks the top-k most frequent keys without keeping the long tail, which is how you separate the handful of stable names from the flood of one off ones. The reservoir keeps a few real samples around for reporting.

With those numbers, classification is a short piece of arithmetic. A field counts as stable if it shows up in at least half the events (the stable_threshold). The number of dynamic keys is whatever the distinct count has that the stable set does not. If that dynamic remainder is large enough, the path is a map:

// event frequency of each heavy hitter, keep the stable ones
let stable: Vec<_> = heavy_hitters
    .top_k(capacity)
    .filter(|f| f.event_frequency >= config.stable_threshold) // default 0.5
    .collect();

let unique = cardinality.estimate();              // HyperLogLog
let dynamic_count = unique.saturating_sub(stable.len());
let has_dynamic_fields = dynamic_count >= config.min_dynamic_fields; // default 5

A few details that matter in practice, all of them config knobs with the defaults shown:

• It waits. No classification happens before min_events (default 100) at a path. Early data is noise.
• It is confident or it abstains. A confidence score, built from how much data it has seen and how cleanly the stable fields separate from the threshold, has to clear confidence_threshold (default 0.7). Below that, it returns nothing and keeps watching.
• It changes its mind. Keys are not dynamic forever. Every reevaluate_interval events (default 10,000) a path gets reclassified, so a field that turns dynamic, or a map that settles down, gets picked up.

The memory is bounded and small. With the default HyperLogLog precision of 12 and 50-slot frequency and sample structures, it works out to about 10 KB per path, fixed, no matter how many distinct keys stream through. That is the whole point of reaching for sketches instead of a HashSet.

Making the fingerprint stable

Detecting the dynamic map is half the job. The other half is computing a structure fingerprint that does not move when the dynamic keys move. DeltaForge hashes the JSON structure recursively, but when it reaches a path classified as a map, it does not hash the keys. It emits a single marker and hashes one representative value’s shape instead:

if classification.has_dynamic_fields {
    "<MAP>".hash(hasher);
    // hash ONE dynamic value's structure, not the keys
    if let Some((_, val)) = obj.iter().find(|(k, _)| !is_stable(k)) {
        hash_value_adaptive(val, hasher, classifications);
    }
}

So {"sess_abc": {...}} and {"sess_xyz": {...}, "sess_123": {...}} hash to the same thing: an object with a stable id and type, plus a <MAP> of { user_id, started_at }. Same shape, same fingerprint, cache hit. That is the flat green line in the demo above.

What it buys you

From the DeltaForge docs, measured on payloads with high-cardinality keys:

The first row is the headline: the difference between a schema registry that thinks your data mutates on every event and one that correctly reports a single stable shape. The third row is the honest cost: while a path is still in its warmup window, there is real overhead doing the full analysis, and it fades to nothing once structures start repeating and the cache takes over.

Where this is honest about its limits

This is a heuristic built on probabilistic structures, and it is worth being clear about what that means:

• HyperLogLog is an estimate. The distinct-key count carries a small error. That is fine here, because the decision is “roughly how many distinct keys,” not an exact count, and the thresholds have margin.
• The thresholds are tuned, not proven. stable_threshold, min_dynamic_fields, and confidence_threshold are defaults that work well on the payloads I have tested. A pathological shape can fool them, which is why reclassification exists: a wrong call gets revisited rather than baked in forever.
• It is opt in. Schema sensing is off by default. You turn it on for the pipelines where the payloads are worth understanding, and leave it off where the relational schema already tells you everything.

That last point is the real framing. Schema tracking gives you the columns. Schema sensing gives you what is inside them. And the only reason sensing is usable on real data, instead of drowning in session ids, is a few kilobytes of HyperLogLog and SpaceSaving per path deciding, continuously, which keys are structure and which keys are just data.

Next in the series

Reading the log is one hard problem. Delivering it is another. The next post is about what “exactly-once delivery” actually means when a single pipeline fans out to Kafka, Redis, NATS, and S3 at the same time, and why it can only be an honest promise per sink.

If you want to poke at the code: DeltaForge lives at github.com/vnvo/deltaforge (the schema sensing internals are in the schema-sensing crate), and the sketches come from flowstats. It is pre-release and I would genuinely value feedback, especially on payloads it classifies badly.