Why Keys Matter
Every Kafka record can carry an optional key, and the key's primary purpose is to determine which partition the record lands in. Because Kafka only guarantees ordering within a single partition, giving related records the same key (for example, a customer ID or order ID) ensures they are always written to the same partition in the order they were produced, which is essential for any use case that depends on processing events for a given entity in sequence.
Cricket analogy: Like assigning every ball bowled by Jasprit Bumrah to always be recorded under his own over-by-over stats line, ensuring all his deliveries stay grouped together in order.
The Default Partitioning Strategy
When a record has a key, the default partitioner hashes the key bytes (using murmur2) and takes the result modulo the number of partitions to pick a target partition, so the same key always maps to the same partition as long as the partition count doesn't change. When a record has no key (key is null), modern Kafka producers use a sticky partitioner that batches consecutive null-key records onto the same partition for a short window before switching, improving batching efficiency compared to strict round-robin.
Cricket analogy: Like a fixed formula deciding which net a bowler practices in based on their squad number modulo the number of nets available, so the same bowler always lands in the same net.
Custom Partitioners and the Repartitioning Risk
You can implement the Partitioner interface to override the default hashing logic, useful when you need business-specific routing, such as sending all records for a VIP customer tier to a dedicated set of partitions for priority processing. The critical risk to understand is that increasing a topic's partition count changes the modulo divisor, so keys that used to hash to partition 3 out of 6 may now hash to a different partition out of 8, silently breaking the ordering guarantee your application relied on for existing keys, since old and new records for the same key can now land on different partitions.
Cricket analogy: Like a tournament organizer manually overriding the standard net-assignment formula to deliberately group all left-arm bowlers into one specific net for targeted coaching, a custom partitioner overriding default hash logic.
public class VipTierPartitioner implements Partitioner {
@Override
public int partition(String topic, Object key, byte[] keyBytes,
Object value, byte[] valueBytes, Cluster cluster) {
int numPartitions = cluster.partitionCountForTopic(topic);
String customerId = (String) key;
if (isVipCustomer(customerId)) {
// route VIP customers to a reserved priority-processing partition
return 0;
}
// default-style hashing for everyone else across the remaining partitions
return 1 + (Math.abs(Utils.murmur2(keyBytes)) % (numPartitions - 1));
}
@Override public void configure(Map<String, ?> configs) {}
@Override public void close() {}
}Never increase a topic's partition count on a topic where per-key ordering matters, without a migration plan. Records for the same key produced before and after the change can land on different partitions, breaking ordering silently with no error raised. Plan partition counts generously upfront, or migrate via a new topic with a controlled cutover.
- A record's key determines its partition, and ordering is only guaranteed within a partition.
- The default partitioner hashes the key with murmur2 and takes it modulo the partition count.
- Null-key records use a sticky partitioner that batches onto one partition briefly for efficiency.
- Custom partitioners implement the Partitioner interface for business-specific routing.
- Increasing partition count changes the hash-to-partition mapping for existing keys.
- Changing partition count can silently break per-key ordering guarantees for old data.
- Plan partition counts upfront generously to avoid needing to grow them later.
Practice what you learned
1. What determines which partition a keyed record is written to by default?
2. Within what scope does Kafka guarantee message ordering?
3. What risk does increasing a topic's partition count introduce?
4. How are null-key records distributed by modern Kafka producers?
5. Why would a team implement a custom Partitioner?
Was this page helpful?
You May Also Like
Kafka Producers Explained
How Kafka producer clients serialize, batch, and send records to topic partitions, and the configuration knobs that control durability and throughput.
Consumer Groups
How Kafka consumer groups distribute partitions across cooperating consumers to scale reads horizontally, and how rebalancing keeps that assignment correct.
Delivery Guarantees in Kafka
The at-most-once, at-least-once, and exactly-once delivery semantics Kafka supports, and the producer/consumer configuration choices that determine which one you get.
Related Reading
Related Study Notes in DevOps
Browse all study notesNginx Study Notes
DevOps · 30 topics
DevOpsAnsible Study Notes
DevOps · 30 topics
DevOpsAdvanced Kubernetes Study Notes
Kubernetes · 30 topics
DevOpsAdvanced Bash Scripting Study Notes
Bash · 30 topics
DevOpsDocker & Kubernetes Study Notes
YAML · 40 topics
DevOpsCI/CD Tools & Pipelines Study Notes
YAML · 37 topics