Kafka Performance Tuning

Analogy categoryAnalogies

Understanding the Throughput vs Latency Trade-off

Kafka performance tuning is fundamentally a trade-off between throughput and latency, and most configuration knobs push in opposite directions on that spectrum. Batching more records together (higher linger.ms, larger batch.size) improves throughput and compression efficiency because fewer, larger network round-trips and disk writes are needed, but it adds latency because each record waits longer before being sent. Before tuning anything, it's essential to measure with realistic workloads using tools like kafka-producer-perf-test.sh and kafka-consumer-perf-test.sh, because the correct settings for a low-latency trading system differ enormously from those for a high-throughput log aggregation pipeline.

🏏

Cricket analogy: It's like a captain deciding whether to bat for quick singles (low latency, low throughput per over) or wait for boundary opportunities by taking more balls (higher throughput per shot but longer waits), a trade-off every batting order must consciously make.

Producer-Side Tuning

The highest-leverage producer settings are linger.ms, batch.size, compression.type, and buffer.memory. Setting linger.ms to even 5-20ms allows the producer to accumulate many records per partition before sending, dramatically improving throughput with minimal latency cost, and pairing it with compression.type=lz4 or zstd typically shrinks network and disk usage by 2-4x for JSON or text payloads with negligible CPU overhead. For write-heavy workloads, increasing buffer.memory prevents the producer from blocking when brokers can't keep up momentarily, but sustained blocking usually indicates the real bottleneck is on the broker or network side, not the producer's buffer size.

🏏

Cricket analogy: It's like a bowler using a slower ball variation (compression) to deceive the batsman with less effort than bowling flat out, achieving a similar outcome (a wicket) with reduced strain, just as compression reduces network load with modest CPU cost.

properties

# High-throughput producer configuration
linger.ms=20
batch.size=131072
compression.type=zstd
buffer.memory=67108864
acks=all
enable.idempotence=true
max.in.flight.requests.per.connection=5

# Consumer tuning for higher throughput
fetch.min.bytes=65536
fetch.max.wait.ms=500
max.partition.fetch.bytes=2097152
max.poll.records=1000

Consumer and Broker Tuning

On the consumer side, fetch.min.bytes and fetch.max.wait.ms control how the broker batches responses back to the consumer — raising fetch.min.bytes reduces the number of fetch requests at the cost of a small added wait when data is sparse. Increasing max.partition.fetch.bytes and max.poll.records lets each poll() pull more data, improving throughput for consumers that can process records quickly, but too-large values risk long processing times per poll that push against max.poll.interval.ms. On the broker side, the most impactful levers are partition count (more partitions enable more parallelism but increase file handle and replication overhead), num.io.threads and num.network.threads sized to available CPU cores, and ensuring the OS page cache has enough free memory since Kafka relies heavily on it rather than JVM heap for read performance.

🏏

Cricket analogy: It's like a stadium's food vendor deciding how large a batch of snacks to prepare before restocking a stand — too small and they restock too often (low fetch.min.bytes), too large and fans wait too long for fresh stock (high fetch.max.wait.ms).

Because Kafka relies on the OS page cache (not JVM heap) for serving reads efficiently, a common broker tuning mistake is over-allocating JVM heap. Keep the Kafka broker heap around 6GB and let the OS use the remaining RAM as page cache — this typically yields far better read throughput than a larger heap.

Increasing partition count is not free: each partition consumes a file handle per log segment on every replica, adds to controller metadata and leader election time, and increases end-to-end latency for producers waiting on acks from more replicas. Over-partitioning a cluster (thousands of partitions per broker) is a common cause of slow rebalances and controller instability.

Kafka tuning is a throughput-versus-latency trade-off; measure with realistic workloads before changing defaults.
linger.ms and batch.size let the producer accumulate records for fewer, larger, more efficient requests.
compression.type=lz4 or zstd typically cuts network and disk usage significantly for text/JSON payloads.
fetch.min.bytes and fetch.max.wait.ms control how much the broker batches data before responding to a consumer fetch.
max.poll.records and max.partition.fetch.bytes increase per-poll throughput but must stay compatible with max.poll.interval.ms.
Broker performance depends heavily on OS page cache, not large JVM heaps — keep heap modest and let the OS cache logs.
More partitions increase parallelism but add file handle, replication, and controller overhead — avoid over-partitioning.