Kafka
Kafka is a distributed event streaming platform used to publish, store, and process continuous streams of data in real time, forming the backbone of many event-driven and data pipeline architectures.
58 resources across 3 libraries
Glossary Terms(11)
Confluent
Confluent is a data streaming platform built around Apache Kafka, offering a fully managed cloud service and enterprise tooling — schema management, stream pro…
ClickHouse
ClickHouse is an open-source, column-oriented database management system built for online analytical processing (OLAP), capable of executing aggregation and an…
Hadoop
Apache Hadoop is an open-source framework for distributed storage and processing of very large datasets across clusters of commodity servers, using simple prog…
Kafka
Kafka is a distributed event streaming platform used to publish, store, and process continuous streams of data in real time, forming the backbone of many event…
Logstash
Logstash is an open-source, server-side data processing pipeline that ingests data from multiple sources, transforms it, and forwards it to a destination such…
Apache ZooKeeper
Apache ZooKeeper is an open-source distributed coordination service that provides a hierarchical key-value store, consensus, and primitives such as locks and l…
Apache HBase
Apache HBase is an open-source, distributed, column-oriented NoSQL database modeled after Google's Bigtable, built to run on top of the Hadoop Distributed File…
Apache Pinot
Apache Pinot is an open-source, real-time distributed OLAP datastore built to answer analytical queries with very low latency over continuously streaming and b…
Materialize
Materialize is a streaming database that lets users write standard SQL, including joins and aggregations, against continuously changing data and get incrementa…
Rockset
Rockset was a cloud-native, real-time analytics database and search engine designed to run low-latency SQL queries directly on raw semi-structured data such as…
Moleculer
Moleculer is a fast, fault-tolerant Node.js microservices framework providing built-in service discovery, load balancing, request/event-based inter-service com…
Study Notes(32)
Flink with Kafka
Understand how to build reliable, exactly-once streaming pipelines by connecting Apache Flink to Apache Kafka as both source and sink.
RabbitMQ vs Kafka
A practical comparison of RabbitMQ's smart-broker model and Kafka's distributed log, and how to pick the right one for a given workload.
Brokers and the Kafka Cluster
How individual Kafka brokers form a cluster, how partitions are distributed and replicated across them, and how the cluster handles broker failure.
Common Kafka Failure Modes
The most frequent ways Kafka clusters and clients fail in production, and how to recognize and recover from each.
Consumer Groups
How Kafka consumer groups distribute partitions across cooperating consumers to scale reads horizontally, and how rebalancing keeps that assignment correct.
Delivery Guarantees in Kafka
The at-most-once, at-least-once, and exactly-once delivery semantics Kafka supports, and the producer/consumer configuration choices that determine which one y…
Installing and Running Kafka Locally
A practical walkthrough of getting a single-node Kafka broker running on your machine, creating a topic, and producing and consuming your first messages.
Kafka Architecture Overview
A tour of how Kafka's components — producers, brokers, topics, consumers, and the controller — fit together to deliver durable, scalable event streaming.
Kafka Connect Explained
How Kafka Connect standardizes moving data into and out of Kafka using configurable connectors instead of custom integration code.
Kafka Consumers Explained
How Kafka consumer clients poll for records, track their read position with offsets, and deserialize messages back into usable objects.
Idempotent and Transactional Producers
How Kafka producers avoid duplicate writes and coordinate atomic multi-partition commits using idempotence and transactions.
Kafka Interview Questions
A structured walkthrough of common Kafka interview topics, from fundamentals to system-design-level questions, with the reasoning interviewers expect.
Kafka Performance Tuning
Practical tuning levers on the producer, consumer, and broker sides for maximizing Kafka throughput and minimizing latency.
Kafka Producers Explained
How Kafka producer clients serialize, batch, and send records to topic partitions, and the configuration knobs that control durability and throughput.
Kafka Quick Reference
A condensed cheat sheet of essential Kafka CLI commands, key configuration properties, and common formulas for day-to-day operational work.
Kafka Replication and Leaders
How Kafka replicates partitions across brokers and elects leaders to guarantee durability and availability.
Kafka Security Basics
Core mechanisms for securing a Kafka cluster: encryption in transit, authentication, and fine-grained authorization.
Kafka Streams Basics
An introduction to Kafka Streams, the client library for building stateful stream processing applications directly on top of Kafka topics.
Kafka vs Other Message Queues
How Kafka's log-based, partitioned architecture compares to traditional brokers like RabbitMQ, cloud queues like AWS SQS, and newer systems like Apache Pulsar.
Kafka with a Programming Language Client
How to produce and consume Kafka messages from application code using official and community client libraries in Python, Java, Go, and Node.js.
Log Compaction
How Kafka's compaction cleanup policy retains only the latest value per key instead of expiring records by time.
Message Keys and Partitioning
How Kafka uses a record's key to choose a partition, why that determines per-key ordering, and the risks of custom partitioners and changing partition counts.
Monitoring Kafka Clusters
The essential metrics, tools, and alerting strategies for keeping a production Kafka cluster healthy and observable.
Retention Policies in Kafka
How Kafka's time- and size-based retention deletes old log segments, and how it interacts with consumer lag.
Showing 24 of 32.
Interview Questions(15)
What is Change Data Capture (CDC)?
Change Data Capture (CDC) is a technique for identifying and streaming only the rows that changed — inserts, updates, and deletes — in a source database, typic…
What is a Message Queue?
A message queue is a middleware component that lets a producer publish messages which sit durably in a buffer until a consumer pulls and processes them, decoup…
What is the Publish-Subscribe (Pub/Sub) Pattern?
The publish-subscribe (pub/sub) pattern is a messaging model where publishers send messages to a topic without knowing who receives them, and subscribers regis…
How to Design a Distributed Message Queue
A distributed message queue is designed as partitioned, append-only logs replicated across brokers, where producers write to a partition by key, consumers in a…
How to Design an Ad Click Aggregator
An ad click aggregator ingests a high-throughput stream of click events, deduplicates and validates them, then rolls them up into per-ad, per-time-window count…
How Would You Design a Distributed Logging System?
Design a distributed logging system as a pipeline: lightweight agents on each host tail and buffer log lines locally, ship them to a durable, partitioned inges…
How to Design a Log Aggregation System
A log aggregation system collects logs from many hosts via lightweight shippers, ships them through a durable buffer to decouple producers from indexing speed,…
What is a Dead Letter Queue?
A dead letter queue (DLQ) is a separate holding queue where messages are routed after they repeatedly fail to be processed successfully, so a poison message ca…
What is Kappa Architecture and How Does It Differ from Lambda?
Kappa architecture simplifies Lambda architecture by processing all data — historical and real-time — through a single stream-processing pipeline, treating the…
What is Change Data Capture (CDC)?
Change data capture is a technique for detecting and streaming every row-level insert, update, and delete made to a database as an ordered sequence of events,…
Exactly-Once vs At-Least-Once Delivery: What Is the Difference?
At-least-once delivery guarantees a message is delivered one or more times (never lost, but possibly duplicated), while exactly-once delivery guarantees a mess…
What Are Message Ordering Guarantees?
Message ordering guarantees define whether and how consumers are assured to see messages in the same sequence producers sent them, ranging from no ordering at…
What is ZooKeeper Used for in DevOps?
Apache ZooKeeper is a distributed coordination service that provides a highly consistent, hierarchical key-value namespace used by other distributed systems to…
What Is Apache Kafka and How Does It Work?
Apache Kafka is a distributed, durable event-streaming platform that stores records as an ordered, append-only log split into partitions, allowing many indepen…
What Is Event-Driven Architecture in DevOps Systems?
Event-driven architecture is a design style where services communicate by emitting and reacting to events — immutable facts describing something that already h…