Kafka Replication and Leaders

Analogy categoryAnalogies

Why Kafka Replicates Partitions

Every partition in a Kafka topic can have multiple replicas spread across different brokers, controlled by the replication.factor topic setting. One replica is designated the leader, and all producer writes and consumer reads for that partition go through the leader; the remaining replicas are followers that passively copy the leader's log. Replication exists purely for durability and availability: if the broker hosting the leader crashes, a follower that was fully caught up can be promoted to leader with no data loss for acknowledged writes.

🏏

Cricket analogy: Think of a partition's replicas like a cricket team carrying both an opening batsman and a designated backup opener in the squad; if Rohit Sharma gets injured before a Test at Lord's, the backup steps in immediately so the innings doesn't stall, just as a follower replica steps in when the leader broker dies.

Leaders, Followers, and the In-Sync Replica Set

The in-sync replica set (ISR) is the subset of a partition's replicas that have fetched up to the leader's latest offset within replica.lag.time.max.ms. Followers pull data from the leader via fetch requests, the same mechanism consumers use, rather than the leader pushing data out. A replica falls out of the ISR if it stops fetching or falls too far behind, and it can rejoin once it catches back up; only ISR members are eligible to become leader without data loss when min.insync.replicas and acks=all are configured together.

🏏

Cricket analogy: The ISR is like the fielders currently inside the 30-yard circle during powerplay overs — only those close enough and positioned correctly, like Jonty Rhodes at cover, are eligible to make the crucial diving stop the captain relies on.

Leader Election

When a leader fails, the controller broker chooses a new leader from the ISR, preferring the first replica in the ISR list to minimize data loss. If unclean.leader.election.enable is turned on, Kafka may promote an out-of-sync replica when no ISR member is available, trading availability for potential data loss. Producers configured with acks=all only receive an acknowledgment once the message is written to the leader and replicated to all current ISR members, which is what makes acks=all safe against leader failover.

🏏

Cricket analogy: When a captain is ruled out injured before a match, the selectors promote the vice-captain who has been playing every match in the current series, not a player who missed the last three fixtures, just as Kafka prefers an ISR member for leader election.

bash

# Inspect a topic's current leader, replicas, and ISR
kafka-topics.sh --describe --topic orders --bootstrap-server broker1:9092

# Example output
# Topic: orders  Partition: 0  Leader: 2  Replicas: 2,1,3  Isr: 2,1,3
# Topic: orders  Partition: 1  Leader: 1  Replicas: 1,3,2  Isr: 1,3

# Producer config for maximum durability
# acks=all
# min.insync.replicas=2 (set on the topic or broker)

Handling Leader Failures and Preferred Leaders

Kafka periodically runs preferred leader election, moving leadership back to the 'preferred leader' (the first replica listed when the partition was created) after a broker that was leader recovers, to keep leadership balanced across the cluster. Without this rebalancing, leadership can pile up on a few brokers after rolling restarts, causing uneven CPU and network load. Tools like kafka-leader-election.sh or Cruise Control automate this so operators don't have to manually trigger PreferredReplicaLeaderElection after every deployment.

🏏

Cricket analogy: After an injury forces a temporary vice-captain to lead for a few overs, the team reverts captaincy to the original captain once fit, keeping leadership consistent rather than letting it drift permanently, like Kafka's preferred leader election restoring balance.

Setting acks=1 (leader-only acknowledgment) or enabling unclean.leader.election.enable=true trades durability for availability — a leader crash before followers replicate a message, or an unclean election promoting a lagging replica, can silently lose acknowledged data. Use acks=all with min.insync.replicas>=2 for critical data.

Each partition has one leader (handles all reads/writes) and follower replicas that replicate via pull-based fetch requests.
The ISR tracks replicas caught up within replica.lag.time.max.ms; only ISR members are safe leader candidates by default.
acks=all plus min.insync.replicas>=2 guarantees an acknowledged write survives a leader failure without data loss.
unclean.leader.election.enable=true allows promoting an out-of-sync replica during outages, risking data loss for availability.
The controller performs leader election and propagates the new leader via LeaderAndIsrRequest to the cluster.
Preferred leader election periodically rebalances leadership back to each partition's originally assigned first replica.