Kafka
By the Apache Software Foundation
Kafka is a distributed event streaming platform used to publish, store, and process continuous streams of data in real time, forming the backbone of many event-driven and data pipeline architectures.
Definition
Kafka is a distributed event streaming platform used to publish, store, and process continuous streams of data in real time, forming the backbone of many event-driven and data pipeline architectures.
Overview
Kafka was originally built at LinkedIn to handle the enormous volume of activity data and metrics the company generated, and was later open sourced and named after the author Franz Kafka. Its core idea is the log: producers write events to append-only, ordered logs called topics, and any number of independent consumers can read from those logs at their own pace, without producers needing to know who's consuming the data or how quickly. Topics are split into partitions distributed across a cluster of brokers, which is what allows Kafka to scale to very high throughput while preserving ordering guarantees within each partition. Consumers track their own position (offset) in each partition, so multiple consumer groups can independently replay or process the same stream of events for different purposes, from real-time analytics to feeding a data warehouse. Kafka has become a standard building block for event-driven microservices, log aggregation, and real-time data pipelines, often compared with alternatives like RabbitMQ for simpler messaging needs or Apache Flink for stream processing built on top of Kafka topics. Companies wanting managed Kafka infrastructure frequently turn to Confluent, founded by Kafka's original creators, and learners can build hands-on skills with it through SkillVeris's Apache Kafka & Messaging course.
Key Features
- Distributed, append-only log architecture for storing event streams
- Topics partitioned across brokers for horizontal scalability
- Multiple independent consumer groups reading the same stream
- Configurable data retention independent of whether messages were consumed
- High throughput and durability suited to real-time data pipelines
- Strong ordering guarantees within individual partitions
- Ecosystem of connectors for integrating with databases and other systems
- Foundation for stream processing frameworks built on top of it