Source Connectors: Getting Data Into Kafka
A source connector's job is to detect changes in an external system and publish them as records into one or more Kafka topics, and the mechanism for detecting those changes varies widely by connector: the Debezium PostgreSQL connector reads the database's write-ahead log to capture every row-level insert, update, and delete as a change-data-capture (CDC) event, while the JDBC source connector instead periodically polls a table with a query, using either an incrementing column or a timestamp column to find new rows since the last poll. CDC-based connectors capture every change with low latency and no polling overhead, while polling-based connectors are simpler to reason about but can miss rapid intermediate updates and add query load to the source database.
Cricket analogy: CDC is like a stump microphone catching every single sound event in real time during play, while polling is like a scorer walking out to check the board every over, both work, but the mic misses nothing while the scorer might miss something that changed and reverted between checks.
Sink Connectors: Getting Data Out of Kafka
A sink connector consumes records from one or more Kafka topics and writes them into a target system, and the interesting design decisions are usually about how it handles delivery semantics and target-side schema. The Elasticsearch sink connector, for instance, can be configured to use a record's key as the document ID, which makes writes idempotent (replaying the same record just overwrites the same document rather than duplicating it), while the S3 sink connector instead batches records into files, flushing based on record count, time, or size thresholds, since object storage rewards fewer, larger files over many tiny ones.
Cricket analogy: Using a record key as a document ID for idempotency is like re-entering a run correction under the same ball number, it overwrites that specific ball's record instead of adding a phantom extra run to the total.
Delivery Semantics and Idempotency
Kafka Connect provides at-least-once delivery by default, meaning a worker crash or rebalance can cause some records to be reprocessed and written to the sink again, so a well-designed sink connector must handle duplicates gracefully, typically through idempotent writes keyed on a unique identifier, as opposed to blindly appending every record. Newer connector versions increasingly support exactly-once semantics for sinks (using Kafka's transactional consumer offset commits alongside the target write) or for sources (using the same transactional producer machinery Kafka Streams uses), but this support is connector-specific and must be explicitly verified rather than assumed.
Cricket analogy: At-least-once delivery is like a scorer re-announcing a boundary over the PA system if they're unsure the first announcement was heard, better to repeat than miss it, but the scoreboard operator must be smart enough not to add the runs twice.
{
"name": "orders-es-sink",
"config": {
"connector.class": "io.confluent.connect.elasticsearch.ElasticsearchSinkConnector",
"connection.url": "http://elasticsearch:9200",
"topics": "shop.public.orders",
"key.ignore": "false",
"schema.ignore": "true",
"behavior.on.null.values": "delete",
"tasks.max": "3"
}
}
Setting key.ignore to false tells the Elasticsearch sink to use each Kafka record's key as the document _id, which is what makes reprocessing after a rebalance safe: the same record just overwrites the same document.
Polling-based source connectors (like the plain JDBC source without CDC) can miss updates that happen and revert between poll intervals, and they add repeated query load to the source database; for high-change-rate tables, a log-based CDC connector is almost always the better choice.
- Source connectors detect external changes and publish them into Kafka; the two dominant strategies are log-based CDC and periodic polling.
- CDC connectors like Debezium read a database's write-ahead log for low-latency, complete change capture.
- Polling connectors are simpler but can miss rapid intermediate changes and add query load.
- Sink connectors write Kafka records to external systems, often batching for efficiency (as with S3).
- Kafka Connect defaults to at-least-once delivery, so sink connectors need idempotent writes to handle duplicates.
- Using a record's key as a target-system identifier (like an Elasticsearch document ID) is a common idempotency pattern.
- Exactly-once support exists in some connectors but is connector-specific and must be verified, not assumed.
Practice what you learned
1. What is the primary advantage of a log-based CDC source connector like Debezium over a polling-based JDBC source connector?
2. Why does Kafka Connect's default at-least-once delivery matter for sink connector design?
3. In the Elasticsearch sink connector, what does setting key.ignore to false accomplish?
4. Why does the S3 sink connector batch records into files rather than writing one object per record?
Was this page helpful?
You May Also Like
Kafka Connect Explained
How Kafka Connect standardizes moving data into and out of Kafka using configurable connectors instead of custom integration code.
Schema Registry and Avro
How the Schema Registry and Avro serialization give Kafka's opaque byte messages an enforced, evolvable structure across producers and consumers.
Stream Processing Concepts
Core concepts behind stream processing: event time versus processing time, windowing strategies, watermarks, and time-bounded joins.
Related Reading
Related Study Notes in DevOps
Browse all study notesNginx Study Notes
DevOps · 30 topics
DevOpsAnsible Study Notes
DevOps · 30 topics
DevOpsAdvanced Kubernetes Study Notes
Kubernetes · 30 topics
DevOpsAdvanced Bash Scripting Study Notes
Bash · 30 topics
DevOpsDocker & Kubernetes Study Notes
YAML · 40 topics
DevOpsCI/CD Tools & Pipelines Study Notes
YAML · 37 topics