Scaling Kafka Clusters

Analogy categoryAnalogies

Scaling Out by Adding Brokers

Unlike a stateless web service, adding a new Kafka broker to a cluster does not automatically rebalance existing data onto it; a freshly joined broker sits idle with zero partitions until an administrator explicitly triggers a partition reassignment using kafka-reassign-partitions.sh, because Kafka has no automatic partition migration for existing topics by default. This is a deliberate design choice: automatic rebalancing could saturate network bandwidth unpredictably during peak traffic if triggered without operator control.

🏏

Cricket analogy: It's like bringing a new net bowler into a franchise's training squad; they don't automatically get rotated into practice sessions with the batsmen, the coach has to explicitly schedule them in, because auto-assigning could disrupt a carefully planned training load for existing bowlers.

Partition reassignment works by generating a JSON plan mapping each partition to its new set of broker replicas, then executing it, during which Kafka streams the actual log segment bytes for each moved replica from the source broker to the destination broker over the network, competing with live production traffic unless throttled. The --throttle flag on kafka-reassign-partitions.sh caps this replication bandwidth so a large rebalance doesn't starve real producer and consumer traffic during business hours.

🏏

Cricket analogy: It's like relocating an entire team's training equipment and archives from one city's facility to another mid-season; you throttle the moving trucks to specific off-peak hours so the transfer doesn't clog the roads the team actually needs for match-day travel.

bash

# Generate a reassignment plan moving partitions to include new broker 4
cat > topics-to-move.json <<EOF
{"topics": [{"topic": "orders"}], "version": 1}
EOF

kafka-reassign-partitions.sh --bootstrap-server broker1:9092 \
  --topics-to-move-json-file topics-to-move.json \
  --broker-list "1,2,3,4" --generate > reassignment-plan.json

# Execute with a throttle to protect live production traffic
kafka-reassign-partitions.sh --bootstrap-server broker1:9092 \
  --reassignment-json-file reassignment-plan.json \
  --throttle 50000000 --execute

# Verify progress and completion
kafka-reassign-partitions.sh --bootstrap-server broker1:9092 \
  --reassignment-json-file reassignment-plan.json --verify

The Cruise Control open-source tool from LinkedIn automates partition reassignment planning by analyzing broker CPU, disk, and network load to generate balanced reassignment plans, removing the guesswork of manually deciding which partitions to move where.

Choosing Partition Count and Scaling Consumers

Partition count is the primary lever for parallelism in Kafka: a topic with 12 partitions can be consumed by at most 12 consumer instances within a single consumer group simultaneously, since each partition is assigned to exactly one consumer in the group at a time, so under-provisioning partitions caps your maximum consumption throughput regardless of how many consumer instances you deploy. However, partition count cannot be easily decreased later without deleting and recreating the topic, so teams typically over-provision moderately upfront based on projected peak throughput.

🏏

Cricket analogy: It's like a stadium having a fixed number of entry gates; even if you hire more ticket-checking staff than gates, throughput is capped by the gate count, just as adding more consumers beyond the partition count doesn't increase parallel processing capacity.

A common capacity planning approach is to estimate target throughput in MB/s, divide by the sustainable throughput of a single partition (commonly benchmarked around 10-20 MB/s per partition depending on hardware and replication factor), and add headroom for future growth, since increasing partition count later requires either accepting a brief reassignment or, if reducing partitions is needed, recreating the topic entirely which breaks key-based ordering guarantees for existing data.

🏏

Cricket analogy: It's like a stadium architect estimating expected match-day attendance and dividing by each turnstile's processing rate to decide how many turnstiles to build, adding extra capacity for a potential World Cup final rather than just an average league match.

Reducing partition count on an existing topic is not supported directly; the only path is deleting and recreating the topic, which loses all existing data and, more subtly, breaks key-based partitioning guarantees, since the same key will now hash to a different partition, potentially reordering related events for downstream consumers.

Adding a broker to a cluster does not automatically rebalance existing partitions onto it; reassignment must be triggered manually.
kafka-reassign-partitions.sh generates and executes partition reassignment plans, moving log segment data between brokers.
The --throttle flag caps reassignment bandwidth to avoid starving live production traffic during a rebalance.
Cruise Control automates reassignment planning by analyzing broker resource utilization across the cluster.
Partition count is the hard ceiling on consumer parallelism within a single consumer group.
Capacity planning divides target throughput by sustainable per-partition throughput, with headroom for future growth.
Reducing partition count requires deleting and recreating the topic, which breaks key-based ordering guarantees.