Kafka Architecture Overview

Analogy categoryAnalogies

A Kafka deployment is a distributed system made up of several cooperating pieces: producer clients that publish records, a cluster of broker servers that store and serve those records, consumer clients that read them, and a metadata/coordination layer (historically ZooKeeper, now the built-in KRaft controller quorum) that tracks cluster membership, topic configuration, and partition leadership. Understanding how these pieces interact, especially how writes are replicated and how leadership is assigned per partition, is essential to reasoning about Kafka's durability and availability guarantees.

🏏

Cricket analogy: A Kafka cluster is like an IPL franchise: the producers are the players scoring runs, the brokers are the different squads (Mumbai, Chennai) holding the data, and the controller is like the BCCI office assigning which team captain (partition leader) is authoritative for a given match.

Brokers, Replication, and Leaders

Each partition of a topic is assigned a replication factor (commonly 3 in production), meaning its data is copied across multiple brokers; exactly one replica is elected the leader and handles all reads and writes for that partition, while the others are followers that continuously fetch from the leader to stay in sync. The set of followers that are sufficiently caught-up to the leader is called the in-sync replica set (ISR), and Kafka only considers a write 'committed' once it's been acknowledged according to the producer's acks setting and, for acks=all, replicated to every member of the ISR, which is what protects against data loss if the leader broker fails.

🏏

Cricket analogy: A partition leader is like the on-strike batter: only they can 'produce' runs on a given ball, while the non-striker (a follower replica) mirrors position but isn't actively facing deliveries, ready to take over strike (leadership) if needed.

Metadata and Coordination: ZooKeeper to KRaft

For most of Kafka's history, cluster metadata (broker list, topic configs, partition leadership, ACLs) was stored in an external Apache ZooKeeper ensemble, and one broker acted as the 'controller' by watching ZooKeeper for changes and propagating leadership decisions to the rest of the cluster. Since KIP-500 and Kafka 3.x/4.x, ZooKeeper has been replaced by KRaft (Kafka Raft), where a small quorum of Kafka brokers themselves (running in 'controller' mode) uses the Raft consensus protocol to store this metadata, simplifying operations by removing an entire external dependency and improving controller failover speed and partition-count scalability.

🏏

Cricket analogy: ZooKeeper was like an external match referee's panel that every team had to consult for rule decisions; KRaft is like the umpires themselves reaching consensus on the field directly, removing the need for a separate off-field panel and speeding up decisions.

properties

# server.properties (KRaft mode) - key architecture-relevant settings
process.roles=broker,controller
node.id=1
controller.quorum.voters=1@localhost:9093,2@localhost:9094,3@localhost:9095
listeners=PLAINTEXT://:9092,CONTROLLER://:9093
controller.listener.names=CONTROLLER

# Replication settings that matter for durability
default.replication.factor=3
min.insync.replicas=2
unclean.leader.election.enable=false

Client-Side Architecture: Producers and Consumers

On the client side, a producer's key job is to serialize records, decide (via a partitioner) which partition each record goes to, batch records for efficiency, and handle retries and acknowledgments per its acks configuration; a consumer's key job is to poll assigned partitions, track its own read position (the offset) for each partition, and periodically commit those offsets so it can resume correctly after a restart. Kafka deliberately keeps brokers simple (they don't track which consumer has read what beyond storing committed offsets), pushing more intelligence to the client libraries, which is a big part of why a Kafka broker can serve many consumer groups reading the same topic at wildly different speeds without extra broker-side bookkeeping.

🏏

Cricket analogy: A producer is like a batsman deciding which gap in the field (partition) to hit the ball into, while a consumer is like a fielder tracking exactly how many balls of the over they've already fielded (their offset), independent of what other fielders are doing.

The Kafka controller (whether ZooKeeper-elected historically or a KRaft quorum leader today) is responsible for detecting broker failures and reassigning partition leadership. When a leader broker goes down, the controller promotes an in-sync replica to leader, typically within seconds, which is why properly configured replication is central to Kafka's high-availability story.

Setting unclean.leader.election.enable=true allows an out-of-sync replica to become leader during an outage to preserve availability, but this can silently lose committed data because the newly elected leader may be missing recent writes. For most production systems handling important data, leave this disabled (the default in modern Kafka) and instead ensure min.insync.replicas is tuned appropriately.

A Kafka cluster consists of producers, brokers, consumers, and a metadata/coordination layer.
Each partition has one leader replica handling reads/writes and follower replicas that stay in sync.
The in-sync replica set (ISR) determines what counts as a 'committed' write under acks=all.
KRaft replaced ZooKeeper as of KIP-500, using Raft consensus among controller-role brokers instead of an external service.
The controller detects broker failures and promotes an in-sync follower to leader during outages.
Producers handle partitioning, batching, and retries; consumers track and commit their own offsets.
Brokers stay simple by delegating offset tracking to consumer clients, enabling many independent consumer groups.