Apache Kafka
By the Apache Software Foundation
Apache Kafka is an open-source distributed event streaming platform used to publish, subscribe to, store, and process high-throughput streams of records in real time, forming the backbone of many event-driven architectures.
Definition
Apache Kafka is an open-source distributed event streaming platform used to publish, subscribe to, store, and process high-throughput streams of records in real time, forming the backbone of many event-driven architectures.
Overview
Kafka began life inside LinkedIn in 2011 to solve the company's internal problem of moving huge volumes of activity and metrics data reliably between systems. It was open-sourced and later donated to the Apache Software Foundation, where it has since become the de facto standard for event streaming infrastructure. At its core, Kafka is a distributed, append-only commit log. Producers write records to named topics, which are split into partitions spread across a cluster of brokers for scalability and fault tolerance; consumers (organized into consumer groups) read those partitions independently and at their own pace, with records retained for a configurable period rather than deleted on read. Historically Kafka relied on ZooKeeper for cluster coordination, but modern versions use a built-in KRaft consensus protocol instead. Kafka is typically deployed on Docker and orchestrated with Kubernetes in production, and its ecosystem includes Kafka Connect for integrating external systems and Kafka Streams for lightweight in-application processing. Kafka sits at the center of many modern data platforms: it decouples microservices, feeds real-time analytics engines such as Apache Spark and Apache Flink, and competes with alternatives like Apache Pulsar and RabbitMQ for different latency and delivery-guarantee tradeoffs. Teams building on Kafka professionally can go deeper with the Apache Kafka & Messaging course.
Key Features
- Distributed, partitioned commit log for durable event storage
- High-throughput, low-latency publish/subscribe messaging
- Consumer groups for parallel, independently-paced reads
- Configurable log retention decoupled from message consumption
- Kafka Connect for integrating databases and external systems
- Kafka Streams for lightweight stream processing without a separate cluster
- KRaft mode for cluster coordination without ZooKeeper
- Built-in replication for fault tolerance across brokers
Use Cases
History
Apache Kafka is a distributed event-streaming platform for building real-time data pipelines and streaming applications. It was originally developed at LinkedIn by Jay Kreps, Neha Narkhede, and Jun Rao to handle the company's enormous volume of real-time event data — activity tracking, metrics, and log aggregation — that existing systems handled inefficiently. Kafka was deployed internally at LinkedIn in early 2011, open-sourced in 2011, and became a top-level Apache Software Foundation project in 2012. Its design as a durable, partitioned, replayable commit log made it a backbone for modern data infrastructure. Kafka's core creators later left LinkedIn to found Confluent, building a commercial ecosystem around it.
Sources
- Apache Kafka — official website · as of 2026-07-17
- Confluent Developer — "Jay Kreps on the Last 10 Years of Apache Kafka" · as of 2026-07-17