Brokers and the Kafka Cluster

Analogy categoryAnalogies

A Kafka broker is a single server process that stores partition data on local disk, serves produce and fetch requests from clients, and participates in replication for the partitions it hosts; a Kafka cluster is simply a group of brokers, each identified by a unique broker.id, working together so that the failure of any single broker doesn't take down the whole system. Rather than storing an entire topic on one machine, Kafka spreads a topic's partitions across the brokers in the cluster (and replicates each partition to multiple brokers), which is what allows both storage capacity and throughput to scale by adding more brokers.

🏏

Cricket analogy: A broker is like one venue hosting part of a multi-city IPL season; the cluster is the whole league of venues working together, so that if one stadium becomes unusable, matches (partitions) hosted elsewhere continue undisturbed.

Partition Placement and Rack Awareness

When a topic is created, Kafka's default partition assignment strategy spreads leaders roughly evenly across brokers and, if rack.id is configured on each broker, tries to place a partition's replicas across different racks (or cloud availability zones) so that the loss of an entire rack doesn't take out every replica of a partition simultaneously. Operators can also reassign partitions manually with the kafka-reassign-partitions.sh tool, for example when decommissioning a broker, rebalancing load after adding new brokers, or moving hot partitions off an overloaded broker, and Kafka streams the data for the moved replicas in the background without interrupting reads or writes to that partition.

🏏

Cricket analogy: Rack awareness is like ensuring a franchise's backup wicketkeeper and star bowler don't both live in the same flood-prone neighborhood, so a single local disaster (rack failure) doesn't remove both key players (replicas) from availability at once.

Scaling and Handling Broker Failure

When a broker crashes or becomes unreachable, the controller detects the failure via missed heartbeats, marks the broker's hosted partitions as leaderless, and promotes an in-sync replica on a healthy broker to leader for each affected partition, typically restoring full read/write availability within seconds; clients using the modern Kafka protocol automatically discover the new leader and redirect traffic without manual intervention. Scaling out the cluster by adding a new broker doesn't automatically move existing partitions onto it, existing partitions stay where they are unless an operator explicitly triggers a reassignment, which is an important operational detail because simply adding hardware doesn't by itself relieve load on already-overloaded brokers.

🏏

Cricket analogy: When a broker fails, it's like a fielder suddenly leaving the ground injured mid-over; the captain (controller) immediately repositions a nearby fielder (in-sync replica) to cover that spot so the game continues without a gap, and the umpire (client) automatically directs the next ball accordingly.

bash

# Check under-replicated partitions across the cluster (a key health signal)
bin/kafka-topics.sh --describe --bootstrap-server localhost:9092 --under-replicated-partitions

# Generate a reassignment plan to move partitions of 'orders' onto brokers 4 and 5
bin/kafka-reassign-partitions.sh --bootstrap-server localhost:9092 \
  --topics-to-move-json-file topics-to-move.json --broker-list "4,5" --generate

# Execute a previously generated reassignment plan
bin/kafka-reassign-partitions.sh --bootstrap-server localhost:9092 \
  --reassignment-json-file reassignment.json --execute

# Verify reassignment progress
bin/kafka-reassign-partitions.sh --bootstrap-server localhost:9092 \
  --reassignment-json-file reassignment.json --verify

The number of under-replicated partitions is one of the most important Kafka cluster health metrics to alert on. A sustained non-zero value means some partitions have fewer in-sync replicas than configured, indicating a struggling broker, network partition, or disk I/O bottleneck that reduces the cluster's fault tolerance until resolved.

Adding new brokers to a cluster does not automatically rebalance existing partitions onto them. Without an explicit reassignment (via kafka-reassign-partitions.sh or a tool like Cruise Control), the new brokers sit nearly empty while older brokers remain just as loaded as before, a common surprise for teams expecting 'just add a broker' to relieve capacity pressure immediately.

A broker is a single Kafka server storing partitions and serving client requests; a cluster is many brokers working together.
Topics are spread across brokers by partition, and each partition is replicated to multiple brokers for fault tolerance.
Rack awareness places replicas of a partition across different racks or availability zones to survive a rack-level failure.
The controller detects broker failure via missed heartbeats and promotes an in-sync replica to leader within seconds.
Clients automatically discover new partition leaders after failover without manual reconfiguration.
Adding a new broker does not automatically move existing partitions onto it; reassignment must be triggered explicitly.
Under-replicated partitions are a critical cluster health metric indicating reduced fault tolerance.