The Three Delivery Semantics
Kafka supports three delivery semantics that describe what happens to a message under failure conditions: at-most-once (a message may be lost but is never duplicated), at-least-once (a message is never lost but may be delivered or processed more than once), and exactly-once semantics, or EOS (a message is processed once and only once, even across failures). Which semantic you get is not automatic; it's determined by specific combinations of producer acks/idempotence settings and consumer commit strategy, and Kafka defaults to at-least-once unless you explicitly configure otherwise.
Cricket analogy: Like the difference between a fielder taking one clean attempt at a run-out throw (at-most-once, might miss), throwing repeatedly until it lands even risking overthrows (at-least-once), or a DRS-reviewed throw guaranteed correct exactly once (exactly-once).
At-Most-Once and At-Least-Once in Practice
At-most-once typically arises from a consumer committing offsets before processing completes, or a producer using acks=0/acks=1 without retries: if a failure happens after the commit or send but before the effect lands, that data is simply gone with no error surfaced. At-least-once, Kafka's default posture, arises from producer retries (which can create duplicate writes if idempotence is off) plus consumers committing offsets only after processing succeeds; a crash between processing and committing means the same records get reprocessed on restart, so downstream logic must tolerate duplicates, typically via idempotent writes keyed on a unique record ID.
Cricket analogy: Like a scorer who never re-checks a boundary call and simply moves on if unsure (at-most-once, data loss risk), versus one who re-confirms with the umpire every time, occasionally recording the same boundary twice if radios overlap (at-least-once, duplicate risk).
Exactly-Once Semantics
Kafka achieves exactly-once semantics for read-process-write pipelines (consume from one topic, transform, produce to another) by combining the idempotent producer (which eliminates duplicate writes from retries using producer IDs and sequence numbers) with Kafka transactions: the producer is configured with transactional.id, wraps its work in beginTransaction()/commitTransaction(), and calls sendOffsetsToTransaction() to atomically commit both the consumer's input offsets and the produced output records in a single transaction. Downstream consumers set isolation.level=read_committed so they only ever see records from transactions that fully committed, never partial or aborted writes.
Cricket analogy: Like a match requiring both the scorer's book and the official ICC scoring system to transactionally agree on a run before it counts, so a disputed run either fully counts in both records or in neither, similar to Kafka's transactional read-process-write atomically committing both the consumed offset and produced output together.
Properties props = new Properties();
props.put("bootstrap.servers", "broker1:9092");
props.put("transactional.id", "orders-enrichment-tx-1");
props.put("enable.idempotence", "true");
props.put("acks", "all");
KafkaProducer<String, String> producer = new KafkaProducer<>(props);
producer.initTransactions();
ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(500));
try {
producer.beginTransaction();
for (ConsumerRecord<String, String> record : records) {
String enriched = enrich(record.value());
producer.send(new ProducerRecord<>("orders-enriched", record.key(), enriched));
}
Map<TopicPartition, OffsetAndMetadata> offsets = currentOffsets(records);
producer.sendOffsetsToTransaction(offsets, consumer.groupMetadata());
producer.commitTransaction();
} catch (Exception e) {
producer.abortTransaction();
}The idempotent producer has been the default in Apache Kafka since version 3.0 whenever acks is not explicitly set (it defaults to acks=all with idempotence on). Exactly-once transactions go further, requiring an explicit transactional.id and read_committed consumers, and only apply to Kafka-to-Kafka pipelines, not to arbitrary external side effects like calling a REST API.
- At-most-once risks losing data but never duplicates it.
- At-least-once, Kafka's default posture, risks duplicates but never silently loses data.
- Exactly-once semantics guarantee each record is processed once, even across failures.
- The idempotent producer removes duplicate writes caused by producer-side retries.
- Kafka transactions atomically commit both produced output and consumed input offsets.
- isolation.level=read_committed ensures consumers never see aborted or partial transactions.
- EOS as implemented by Kafka transactions applies to Kafka-to-Kafka pipelines, not external systems.
Practice what you learned
1. Which delivery semantic risks losing messages but never produces duplicates?
2. What is Kafka's default delivery posture without extra configuration?
3. What does the idempotent producer prevent specifically?
4. What does sendOffsetsToTransaction() accomplish?
5. What does isolation.level=read_committed do for a consumer?
Was this page helpful?
You May Also Like
Kafka Producers Explained
How Kafka producer clients serialize, batch, and send records to topic partitions, and the configuration knobs that control durability and throughput.
Kafka Consumers Explained
How Kafka consumer clients poll for records, track their read position with offsets, and deserialize messages back into usable objects.
Message Keys and Partitioning
How Kafka uses a record's key to choose a partition, why that determines per-key ordering, and the risks of custom partitioners and changing partition counts.
Related Reading
Related Study Notes in DevOps
Browse all study notesNginx Study Notes
DevOps · 30 topics
DevOpsAnsible Study Notes
DevOps · 30 topics
DevOpsAdvanced Kubernetes Study Notes
Kubernetes · 30 topics
DevOpsAdvanced Bash Scripting Study Notes
Bash · 30 topics
DevOpsDocker & Kubernetes Study Notes
YAML · 40 topics
DevOpsCI/CD Tools & Pipelines Study Notes
YAML · 37 topics