Change Data Capture (CDC)
Change Data Capture (CDC) is a set of techniques for detecting and capturing row-level inserts, updates, and deletes in a source database as they happen, so those changes can be propagated to downstream systems in near real time.
Definition
Change Data Capture (CDC) is a set of techniques for detecting and capturing row-level inserts, updates, and deletes in a source database as they happen, so those changes can be propagated to downstream systems in near real time.
Overview
Traditional batch integration reloads entire tables on a schedule, which wastes compute and leaves downstream systems stale for hours. CDC instead watches a database's transaction log (or uses triggers, timestamps, or periodic diffing) and emits a stream of change events the moment a row is inserted, updated, or deleted, dramatically reducing latency and load. Log-based CDC is the most common production approach: tools like Debezium tail the write-ahead log of a source database and publish each change as an event, typically onto a message bus such as Kafka for downstream consumers to subscribe to. This decouples the source system from consumers and lets many downstream services react to the same stream of changes independently. CDC is a foundational building block for keeping a data warehouse or data lake in sync with operational databases without full reloads, for feeding ETL and ELT pipelines incrementally, for cache invalidation, for audit trails, and for driving database replication between heterogeneous systems. Modern data platforms like the Apache Kafka & Messaging course cover CDC as a core pattern for building event-driven architectures where every state change becomes a first-class, replayable event. Because CDC captures every intermediate state change rather than just a final snapshot, it also underpins data lineage tracking and enables patterns like event sourcing, where an application's state is reconstructed by replaying its full history of changes.
Key Concepts
- Log-based capture that reads database transaction logs without impacting source performance
- Near real-time propagation of inserts, updates, and deletes to downstream consumers
- Decouples source systems from consumers via an event stream, often built on a message bus
- Supports incremental loading into warehouses and lakes instead of full-table reloads
- Preserves ordering and, in many implementations, exactly-once or at-least-once delivery semantics
- Works across heterogeneous databases (MySQL, PostgreSQL, MongoDB, SQL Server, Oracle)
- Enables event sourcing and audit-trail patterns by capturing every intermediate state
- Can trigger downstream cache invalidation, search-index updates, or microservice notifications