Why Duplicates Happen Without Idempotence
When a producer sends a record and the broker writes it successfully but the acknowledgement is lost on the network before reaching the producer, the producer's retry logic will resend the record, because it has no way of knowing the write actually succeeded. Without any deduplication mechanism at the broker, this retry creates a second, duplicate record in the partition log, silently corrupting exactly-once expectations for downstream consumers.
Cricket analogy: It is like a third umpire's decision being sent to the scorer but the confirmation signal drops, so the scorer, unsure if the run was recorded, adds it again, giving the batsman an extra run on the board that never actually happened.
Kafka's idempotent producer solves this at the broker level: each producer instance is assigned a unique Producer ID (PID), and every batch it sends carries a monotonically increasing sequence number per partition. The broker tracks the last five sequence numbers it has committed per (PID, partition) pair, and if it receives a batch whose sequence number it has already seen, it discards the duplicate and returns the original success acknowledgement rather than writing the record again.
Cricket analogy: It resembles how a scoring app assigns each ball bowled in an over a sequential number, so if the same ball's data packet arrives twice due to a network hiccup, the app recognizes ball number 4 of over 12 was already recorded and ignores the repeat.
Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "broker1:9092,broker2:9092");
props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true);
props.put(ProducerConfig.ACKS_CONFIG, "all");
props.put(ProducerConfig.RETRIES_CONFIG, Integer.MAX_VALUE);
props.put(ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, 5);
// Transactional producer for exactly-once writes across partitions/topics
props.put(ProducerConfig.TRANSACTIONAL_ID_CONFIG, "order-processor-1");
KafkaProducer<String, String> producer = new KafkaProducer<>(props);
producer.initTransactions();
try {
producer.beginTransaction();
producer.send(new ProducerRecord<>("orders", key, orderJson));
producer.send(new ProducerRecord<>("inventory-updates", key, inventoryJson));
producer.commitTransaction();
} catch (KafkaException e) {
producer.abortTransaction();
}Setting enable.idempotence=true automatically forces acks=all, retries to a high value, and max.in.flight.requests.per.connection to at most 5, guaranteeing ordering is preserved even with pipelined retries.
Transactions Across Partitions and Topics
Idempotence alone only guarantees no duplicates within a single partition from a single producer session; it says nothing about atomicity across multiple partitions or topics. Kafka transactions extend this by letting a producer group multiple sends, potentially spanning many partitions and topics, into a single atomic unit using beginTransaction, commitTransaction, and abortTransaction, coordinated by a transaction coordinator broker that writes control markers to each involved partition.
Cricket analogy: It's like a DRS review that must simultaneously update the ball-tracking graphic, the umpire's decision, and the scoreboard all together; if any one system fails to update, the whole review is treated as void rather than leaving the scoreboard out of sync with the decision.
Consumers reading transactionally written data must set isolation.level to read_committed to see only records from committed transactions and skip records from aborted ones; the default read_uncommitted setting exposes consumers to every write regardless of transaction outcome, which defeats the purpose of using transactions for exactly-once pipelines like Kafka Streams' EOS mode.
Cricket analogy: It's similar to a stadium's official scoreboard only updating after the third umpire confirms a decision, so spectators watching the confirmed board never see a run that was later overturned, unlike someone watching the raw, unconfirmed feed on their phone.
A transactional producer that crashes mid-transaction leaves the transaction open until transaction.timeout.ms elapses, after which the new producer instance with the same transactional.id will fence the old one via a bumped producer epoch and the coordinator will abort the stale transaction.
- Idempotent producers use a Producer ID plus per-partition sequence numbers so brokers can detect and discard duplicate retries.
- enable.idempotence=true automatically enforces acks=all and bounds in-flight requests to preserve ordering during retries.
- Transactions extend idempotence to atomic, multi-partition and multi-topic writes coordinated by a transaction coordinator broker.
- Consumers must set isolation.level=read_committed to only see data from committed transactions, not aborted ones.
- A stable, unique transactional.id per producer instance allows fencing of zombie producers via producer epoch bumps.
- Kafka Streams' exactly-once semantics (EOS) mode is built directly on top of transactional producers.
- Idempotence alone prevents duplicates within one partition; transactions are required for cross-partition atomicity.
Practice what you learned
1. What two pieces of metadata does an idempotent producer attach to detect duplicate batches?
2. What consumer configuration is required to only read committed transactional data?
3. What happens when enable.idempotence is set to true?
4. What component coordinates a Kafka transaction across multiple partitions?
5. Why must a transactional.id remain stable across producer restarts?
Was this page helpful?
You May Also Like
Monitoring Kafka Clusters
The essential metrics, tools, and alerting strategies for keeping a production Kafka cluster healthy and observable.
Common Kafka Failure Modes
The most frequent ways Kafka clusters and clients fail in production, and how to recognize and recover from each.
Kafka Security Basics
Core mechanisms for securing a Kafka cluster: encryption in transit, authentication, and fine-grained authorization.
Related Reading
Related Study Notes in DevOps
Browse all study notesNginx Study Notes
DevOps · 30 topics
DevOpsAnsible Study Notes
DevOps · 30 topics
DevOpsAdvanced Kubernetes Study Notes
Kubernetes · 30 topics
DevOpsAdvanced Bash Scripting Study Notes
Bash · 30 topics
DevOpsDocker & Kubernetes Study Notes
YAML · 40 topics
DevOpsCI/CD Tools & Pipelines Study Notes
YAML · 37 topics