Apache Kafka Study Notes

Brokers and the Kafka Cluster

How individual Kafka brokers form a cluster, how partitions are distributed and replicated across them, and how the cluster handles broker failure.

Common Kafka Failure Modes

The most frequent ways Kafka clusters and clients fail in production, and how to recognize and recover from each.

Consumer Groups

How Kafka consumer groups distribute partitions across cooperating consumers to scale reads horizontally, and how rebalancing keeps that assignment correct.

Delivery Guarantees in Kafka

The at-most-once, at-least-once, and exactly-once delivery semantics Kafka supports, and the producer/consumer configuration choices that determine which one y…

Installing and Running Kafka Locally

A practical walkthrough of getting a single-node Kafka broker running on your machine, creating a topic, and producing and consuming your first messages.

Kafka Architecture Overview

A tour of how Kafka's components — producers, brokers, topics, consumers, and the controller — fit together to deliver durable, scalable event streaming.

Kafka Connect Explained

How Kafka Connect standardizes moving data into and out of Kafka using configurable connectors instead of custom integration code.

Kafka Consumers Explained

How Kafka consumer clients poll for records, track their read position with offsets, and deserialize messages back into usable objects.

Idempotent and Transactional Producers

How Kafka producers avoid duplicate writes and coordinate atomic multi-partition commits using idempotence and transactions.

Kafka Interview Questions

A structured walkthrough of common Kafka interview topics, from fundamentals to system-design-level questions, with the reasoning interviewers expect.

Kafka Performance Tuning

Practical tuning levers on the producer, consumer, and broker sides for maximizing Kafka throughput and minimizing latency.

Kafka Producers Explained

How Kafka producer clients serialize, batch, and send records to topic partitions, and the configuration knobs that control durability and throughput.

Kafka Quick Reference

A condensed cheat sheet of essential Kafka CLI commands, key configuration properties, and common formulas for day-to-day operational work.

Kafka Replication and Leaders

How Kafka replicates partitions across brokers and elects leaders to guarantee durability and availability.

Kafka Security Basics

Core mechanisms for securing a Kafka cluster: encryption in transit, authentication, and fine-grained authorization.

Kafka Streams Basics

An introduction to Kafka Streams, the client library for building stateful stream processing applications directly on top of Kafka topics.

Kafka vs Other Message Queues

How Kafka's log-based, partitioned architecture compares to traditional brokers like RabbitMQ, cloud queues like AWS SQS, and newer systems like Apache Pulsar.

Kafka with a Programming Language Client

How to produce and consume Kafka messages from application code using official and community client libraries in Python, Java, Go, and Node.js.

Log Compaction

How Kafka's compaction cleanup policy retains only the latest value per key instead of expiring records by time.

Message Keys and Partitioning

How Kafka uses a record's key to choose a partition, why that determines per-key ordering, and the risks of custom partitioners and changing partition counts.

Monitoring Kafka Clusters

The essential metrics, tools, and alerting strategies for keeping a production Kafka cluster healthy and observable.

Retention Policies in Kafka

How Kafka's time- and size-based retention deletes old log segments, and how it interacts with consumer lag.

Scaling Kafka Clusters

Strategies for growing Kafka capacity: adding brokers, reassigning partitions, and choosing the right partition count upfront.

Schema Registry and Avro

How the Schema Registry and Avro serialization give Kafka's opaque byte messages an enforced, evolvable structure across producers and consumers.