Consumer Groups

A consumer group is a set of consumer instances sharing the same group.id that jointly consume a topic's partitions, with Kafka guaranteeing that each partition is assigned to exactly one consumer within the group at any given time. This lets you scale reads horizontally: adding more consumer instances (up to the number of partitions) increases parallel processing throughput, while multiple independent groups can each read the full topic independently for entirely different purposes.

When a consumer joins or leaves a group (including crashing, or being killed by exceeding max.poll.interval.ms), the group coordinator triggers a rebalance to redistribute partitions among the remaining members. The original 'eager' rebalance protocol revokes every partition from every consumer before reassigning them all, causing a brief full-group pause; the newer cooperative sticky assignor (partition.assignment.strategy=CooperativeStickyAssignor) instead only reassigns the specific partitions that need to move, letting unaffected consumers keep processing without interruption.

A consumer group can never usefully have more active consumers than the topic has partitions, since Kafka assigns each partition to exactly one consumer; extra consumers simply sit idle. For consumers that restart frequently (rolling deploys, pod rescheduling), setting group.instance.id enables static membership, letting a restarting consumer rejoin with its previous partition assignment intact within session.timeout.ms, avoiding an unnecessary rebalance for what is really just a brief, expected restart.