Topics, Partitions, and Offsets

Analogy categoryAnalogies

A topic is Kafka's logical category for a stream of records, such as orders, page-views, or payment-events, and every topic is split into one or more partitions, each of which is an independently ordered, append-only log stored on disk. Kafka guarantees strict ordering only within a single partition, not across an entire topic, and every record within a partition is assigned a sequential integer offset (0, 1, 2, ...) the moment it's appended, which consumers use as the durable bookmark for exactly where they are in that partition's log.

🏏

Cricket analogy: A topic is like the overall 'match', while each partition is like one bowler's individual over sequence: deliveries within an over (partition) are strictly ordered ball 1 through 6 (offsets), but there's no guaranteed cross-bowler ordering across the whole innings.

Why Partitions Matter: Parallelism and Ordering Trade-offs

Partitions are Kafka's primary unit of horizontal scale: because each partition can be hosted on a different broker and read by a different consumer within a consumer group, adding more partitions lets you add more consumers to increase read throughput, up to the point where the number of active consumers equals the number of partitions (beyond that, extra consumers sit idle). The trade-off is that if you need strict ordering across a set of related events, such as all updates for a given customer ID, you must ensure they land in the same partition, which Kafka achieves by default via hashing the record's key, so choosing a good partition key (like customer_id rather than a random UUID) is one of the most consequential early design decisions in a Kafka-based system.

🏏

Cricket analogy: Adding partitions is like adding more bowling attacks to bowl overs in parallel from both ends, speeding up the innings, but if you need every delivery to a specific batsman analyzed in strict order, you must ensure the same 'partition' (analysis feed) always tracks that batsman.

Offsets and Consumer Position Tracking

Every record's offset is a monotonically increasing, partition-local integer assigned by the broker at append time and never reused, even if earlier records are later deleted by retention or compaction; consumers periodically commit the offset of the last record they've successfully processed, typically to Kafka's internal __consumer_offsets topic, so that if a consumer crashes and restarts (or a new consumer takes over during a rebalance), it resumes exactly where the previous one left off rather than reprocessing everything or skipping records. This offset-commit model is also what makes 'replaying' data possible: an operator can manually reset a consumer group's committed offset to an earlier point, or to the very beginning, to reprocess historical events, for example after fixing a bug in the consuming application.

🏏

Cricket analogy: An offset is like the exact ball number in an over a scorer has last logged; if the scorer's laptop crashes, they check the last logged ball number and resume from ball 4 rather than re-scoring balls 1 through 3 or skipping to ball 6.

java

// Producer sending records keyed by customerId so all events for one customer
// land in the same partition, preserving per-customer ordering.
ProducerRecord<String, String> record =
    new ProducerRecord<>("customer-events", customerId, eventJson);
producer.send(record);

// Consumer manually committing offsets after processing, for at-least-once semantics
while (true) {
    ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(500));
    for (ConsumerRecord<String, String> r : records) {
        process(r.value());
    }
    consumer.commitSync(); // commits the offset of the last record processed
}

You can inspect and reset consumer group offsets using the CLI: bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group my-group --describe shows current offsets and lag per partition, while --reset-offsets --to-earliest --execute rewinds a group to replay a topic from the start, a common technique after fixing a downstream processing bug.

Choosing a poor partition key, such as a low-cardinality field like a boolean status flag, causes 'hot partitions' where most records hash to one or two partitions, creating a throughput bottleneck and unbalanced load across brokers, even though the topic technically has many partitions available. Prefer high-cardinality, evenly distributed keys like customer or order IDs.

A topic is a logical stream split into one or more independently ordered partitions.
Kafka guarantees ordering only within a partition, never across an entire topic.
Each record gets a monotonically increasing, partition-local offset at append time.
Partitions are the unit of parallelism; more partitions allow more concurrent consumers.
Records with the same key hash to the same partition by default, preserving per-key ordering.
Consumers commit offsets (often to __consumer_offsets) to resume correctly after a restart.
Offsets can be manually reset to replay historical data, e.g., after fixing a processing bug.