What Log Compaction Solves
Log compaction is an alternative retention strategy, set via log.cleanup.policy=compact, that keeps only the most recent value for each message key rather than deleting records after a time or size limit. This makes compacted topics ideal for representing current state — such as a changelog of the latest account balance per customer ID — where old intermediate values are irrelevant once a newer update for the same key arrives. Kafka's internal __consumer_offsets topic and Kafka Streams' state-store changelog topics both rely on compaction to stay bounded in size while preserving the latest state for every key.
Cricket analogy: A live cricket scoreboard only shows each batsman's current score, not a full ball-by-ball replay, since only the latest run tally per player matters for the state of the match, just as compaction keeps only the newest value per key.
How the Compaction Process Works
Compaction runs in the background via dedicated log cleaner threads that scan each log segment and build an offset map of the latest offset for every key, then rewrite segments keeping only those latest records. A partition becomes eligible for cleaning once its 'dirty ratio' — the proportion of the log written since the last cleaning relative to total size — exceeds min.cleanable.dirty.ratio (default 0.5), balancing CPU cost against how much stale data accumulates. Because compaction is asynchronous, a compacted topic can briefly contain multiple values for the same key until the cleaner catches up.
Cricket analogy: Groundstaff only re-roll and repair the pitch once a threshold of overs have worn it down, not after every single delivery, similar to the cleaner thread only compacting a partition once its dirty ratio crosses the threshold.
Tombstones and Deletion
To delete a key entirely from a compacted topic, producers write a tombstone: a record with the key set and a null value. The cleaner keeps tombstones around for at least delete.retention.ms (default 24 hours) so that consumers who are behind still see the deletion before it's permanently removed, then purges the tombstone once that window passes and no consumer needs it anymore. Deleting too early would let a slow consumer conclude a key still exists when it was actually deleted.
Cricket analogy: When a player is dropped from the squad, the team roster doesn't erase their stats page instantly — it stays visible for a grace period so fans following the old roster still see the update, like a tombstone kept for delete.retention.ms before final removal.
# Create a compacted topic for an account balance changelog
kafka-topics.sh --create --topic account-balances \
--bootstrap-server broker1:9092 \
--partitions 6 --replication-factor 3 \
--config cleanup.policy=compact \
--config min.cleanable.dirty.ratio=0.3 \
--config segment.ms=600000
# Delete a key by writing a tombstone (null value) with the Java producer
# producer.send(new ProducerRecord<>("account-balances", "cust-4821", null));Compaction vs Deletion, and Use Cases
Compaction differs fundamentally from time- or size-based deletion: a deletion policy (log.cleanup.policy=delete) drops whole segments once they age out regardless of key, while compaction is key-aware and never removes the newest record for a key no matter how old it is. Topics can even combine both policies (compact,delete) to compact old segments for space while also enforcing a hard retention ceiling, which is common for changelog topics that still need an eventual cleanup bound.
Cricket analogy: A cricket board keeps a permanent record of each player's highest score ever, no matter how old, while also purging old ground maintenance logs after a year, mirroring how compaction retains latest-per-key forever while delete policy purges by age.
Compacted topics guarantee that the latest value per key is retained forever, but they do NOT guarantee every intermediate update is retained — if you need a full audit trail of every change, use a separate time-retained topic or enable both compact and delete policies carefully.
- cleanup.policy=compact retains only the latest value per key instead of expiring records by age.
- Background cleaner threads compact a partition once its dirty ratio exceeds min.cleanable.dirty.ratio.
- A key is deleted by writing a tombstone (null-value record) for that key.
- Tombstones persist for delete.retention.ms so lagging consumers can still observe the deletion.
- Compacted topics power Kafka internals like __consumer_offsets and Kafka Streams changelog topics.
- cleanup.policy=compact,delete combines key-aware compaction with a hard retention ceiling.
Practice what you learned
1. What does log compaction retain for each key?
2. How does a producer delete a key from a compacted topic?
3. What triggers the log cleaner to compact a partition?
4. Why does Kafka retain tombstones for delete.retention.ms instead of removing them immediately?
5. Which Kafka internal topic relies on compaction to stay bounded in size?
Was this page helpful?
You May Also Like
Retention Policies in Kafka
How Kafka's time- and size-based retention deletes old log segments, and how it interacts with consumer lag.
Kafka Replication and Leaders
How Kafka replicates partitions across brokers and elects leaders to guarantee durability and availability.
The Kafka Controller
The broker role responsible for cluster-wide coordination: leader election, metadata propagation, and admin operations.
Related Reading
Related Study Notes in DevOps
Browse all study notesNginx Study Notes
DevOps · 30 topics
DevOpsAnsible Study Notes
DevOps · 30 topics
DevOpsAdvanced Kubernetes Study Notes
Kubernetes · 30 topics
DevOpsAdvanced Bash Scripting Study Notes
Bash · 30 topics
DevOpsDocker & Kubernetes Study Notes
YAML · 40 topics
DevOpsCI/CD Tools & Pipelines Study Notes
YAML · 37 topics