ZooKeeper vs KRaft

Analogy categoryAnalogies

Why Kafka Needed a Metadata Layer

For most of Kafka's history, cluster metadata — the list of topics, partitions, replica assignments, ACLs, and which broker is currently the controller — was stored in an external Apache ZooKeeper ensemble rather than inside Kafka itself. Brokers watched ZooKeeper znodes for changes and used ZooKeeper's ephemeral nodes and session mechanism for controller election and broker liveness detection, which meant every Kafka cluster required operators to separately deploy, tune, and secure a completely different distributed system with its own quorum, ports, and failure modes.

🏏

Cricket analogy: Historically, a cricket board outsourced ticketing entirely to a third-party vendor with its own servers and staff, meaning every stadium had to integrate with a separate outside system just to know who's supposed to be seated where, like Kafka depending on an external ZooKeeper ensemble for metadata.

How KRaft Replaces ZooKeeper

KRaft (Kafka Raft), introduced via KIP-500 and made the default in Kafka 3.x/4.x, removes the ZooKeeper dependency entirely by storing metadata as an event-sourced log replicated using the Raft consensus protocol among a set of controller nodes, written to an internal topic called __cluster_metadata. Instead of brokers polling and watching znodes, brokers now consume this metadata log directly, much like consumers reading any other Kafka topic, which unifies Kafka's operational model around a single technology (Kafka's own log replication) instead of two.

🏏

Cricket analogy: Instead of outsourcing ticketing, a modern cricket board built ticketing directly into its own existing stadium management platform, so there's only one system to run, mirroring KRaft folding metadata into Kafka's own log replication instead of a separate ZooKeeper system.

Migration and Deployment Modes

KRaft nodes run in one of two modes, set via process.roles: controller-only nodes that participate solely in the Raft quorum and metadata management, or combined broker+controller nodes (process.roles=broker,controller) suitable for smaller clusters. Migrating an existing cluster from ZooKeeper to KRaft is a supported, staged process (KIP-866) involving provisioning a KRaft controller quorum, enabling ZK migration mode so brokers dual-write metadata, then finally cutting brokers over to KRaft-only and decommissioning ZooKeeper — a mistake in following the staged order risks metadata inconsistency.

🏏

Cricket analogy: A small local cricket club has one person who both umpires and manages the scorebook, while a national board has dedicated umpires separate from dedicated scorers, mirroring KRaft's combined mode for small clusters versus dedicated controller nodes for larger ones.

properties

# server.properties for a combined broker+controller KRaft node
process.roles=broker,controller
node.id=1
controller.quorum.voters=1@host1:9093,2@host2:9093,3@host3:9093
listeners=PLAINTEXT://:9092,CONTROLLER://:9093
controller.listener.names=CONTROLLER

Operational and Scalability Differences

Operationally, KRaft removes an entire class of failure modes and tuning knobs tied to running ZooKeeper (session timeouts, jute.maxbuffer sizing, separate TLS/ACL configuration) and dramatically improves controller failover time and metadata scalability, since Raft-based leader election typically completes in a couple of seconds versus ZooKeeper session-timeout-bound elections that could take tens of seconds, and clusters can now support far more partitions because metadata is no longer bottlenecked by ZooKeeper's in-memory znode tree and single-writer model.

🏏

Cricket analogy: Merging ticketing into the stadium's own system means fewer outside vendors to coordinate with, and if there's an outage, the board's own on-call team can fix it directly instead of waiting on a vendor, like KRaft removing an entire external ZooKeeper failure mode.

ZooKeeper mode is deprecated and removed starting in Kafka 4.0 — any cluster still running in ZK mode must plan a KIP-866 migration to KRaft before upgrading, since new major versions no longer ship ZooKeeper support at all.

Historically Kafka stored cluster metadata in an external ZooKeeper ensemble.
KRaft (KIP-500) removes the ZooKeeper dependency by replicating metadata via Raft into an internal __cluster_metadata topic.
KRaft nodes run as controller-only or combined broker+controller, set via process.roles.
Migrating from ZooKeeper to KRaft is a staged, supported process defined by KIP-866.
KRaft offers faster controller failover than ZooKeeper session-timeout-bound elections.
ZooKeeper mode is removed starting in Kafka 4.0, requiring migration before upgrading.