What is Change Data Capture (CDC)?
Learn what Change Data Capture is, how log-based CDC works, and why it beats batch extraction for real-time data sync.
Expected Interview Answer
Change Data Capture (CDC) is a technique for identifying and streaming only the rows that changed — inserts, updates, and deletes — in a source database, typically by reading its transaction log, instead of repeatedly re-extracting the entire dataset.
Traditional batch extraction re-scans a whole table on a schedule, which grows slower and more resource-intensive as the table grows and can miss changes made and reverted between batches. Log-based CDC instead taps directly into the database's write-ahead log or binlog, the same stream the database itself uses for replication, and emits an ordered event for every committed change with minimal load on the source. Those change events are typically published to a message stream (like Kafka) and consumed downstream to keep a data warehouse, cache, or search index continuously in sync with the source, often within seconds rather than hours.
- Captures every change without full-table re-scans
- Low latency — downstream systems stay near real-time in sync
- Minimal load on the source database compared to batch queries
- Preserves an ordered history of inserts, updates, and deletes
AI Mentor Explanation
Think of a stadium's official commentary feed, which announces every single event — a run, a wicket, a boundary — the instant it happens, rather than a reporter walking out every hour to compare the current scoreboard against what it read an hour ago. That hourly comparison approach is slow, misses events that were quickly reversed by a review, and gets more painful as the match drags on. Change data capture is the commentary feed: it taps directly into the stream of events as they are committed, so every change is captured in order with almost no delay.
Step-by-Step Explanation
Step 1
Tap the transaction log
A CDC connector reads the source database's write-ahead log or binlog, the same stream used for replication.
Step 2
Convert log entries to change events
Each committed insert, update, or delete becomes a structured, ordered event.
Step 3
Publish to a stream
Change events are typically published to a message broker like Kafka for downstream consumption.
Step 4
Apply changes downstream
Consumers apply the events to a warehouse, cache, or search index, keeping it continuously synchronized with the source.
What Interviewer Expects
- Clear explanation of log-based CDC versus batch polling
- Understanding of why CDC reduces load on the source database
- Awareness of typical downstream use (sync to warehouse, cache invalidation)
- Mention of ordering and low-latency guarantees CDC provides
Common Mistakes
- Confusing CDC with simple polling on a "last_updated" column
- Not mentioning that CDC reads the transaction log rather than the tables directly
- Forgetting to explain the downside of batch extraction it solves
- Failing to mention a common downstream architecture like Kafka
Best Answer (HR Friendly)
“Change data capture means streaming only the rows that actually changed in a database — inserts, updates, deletes — instead of repeatedly re-scanning the whole table. It reads the database's own transaction log, so downstream systems like a data warehouse or cache can stay nearly real-time in sync with very little extra load on the source.”
Code Example
-- A CDC connector tailing the database's transaction log
-- turns each committed statement into a structured change event, e.g.:
UPDATE Orders SET status = 'shipped' WHERE order_id = 9931;
-- becomes an event like:
-- { "op": "update", "table": "Orders", "before": { "status": "pending" },
-- "after": { "status": "shipped" }, "order_id": 9931, "commit_ts": "..." }
DELETE FROM Orders WHERE order_id = 8820;
-- becomes an event like:
-- { "op": "delete", "table": "Orders", "order_id": 8820, "commit_ts": "..." }
-- Downstream consumers apply these events to keep
-- a warehouse or search index continuously in sync.Follow-up Questions
- What is the difference between log-based CDC and trigger-based CDC?
- How does CDC compare to batch ELT for keeping a warehouse up to date?
- What tools implement log-based CDC (e.g. Debezium)?
- How does CDC handle schema changes in the source table?
MCQ Practice
1. What does log-based CDC typically read from the source database?
Log-based CDC taps the same transaction log the database uses internally for replication, capturing committed changes with minimal source load.
2. What is a key advantage of CDC over periodic batch extraction?
CDC streams individual committed changes as they occur, avoiding expensive full re-scans and missed intermediate changes.
3. CDC events are commonly published to which kind of system for downstream consumption?
CDC connectors typically publish ordered change events to a streaming platform like Kafka, which downstream consumers subscribe to.
Flash Cards
What is Change Data Capture? — A technique for streaming only the changed rows (inserts, updates, deletes) from a source database.
How does log-based CDC work? — It reads the database's transaction log directly, the same stream used for replication.
Why is CDC better than batch polling? — It has lower latency, less source load, and does not miss changes reverted between batch runs.
Where are CDC events commonly published? — To a message stream like Kafka for downstream consumers to process.