Kafka with a Programming Language Client

Analogy categoryAnalogies

Choosing and Using a Kafka Client Library

Apache Kafka does not ship a single universal client; instead the ecosystem provides language-specific libraries such as the official kafka-clients Java library, confluent-kafka-python (built on librdkafka), segmentio/kafka-go for Go, and kafkajs for Node.js. All of them implement the same binary wire protocol, so a producer written in Python and a consumer written in Java can read and write the same topic without any translation layer. Choosing between them usually comes down to language fit, how completely the library implements features like transactions and exactly-once semantics, and whether it wraps the battle-tested C library librdkafka or reimplements the protocol natively.

🏏

Cricket analogy: Just as a franchise can field an all-rounder from Mumbai Indians and a spinner from Chennai Super Kings on the same national team because both were trained under BCCI's unified rulebook, producers and consumers in different languages interoperate because they follow the same Kafka wire protocol.

Producing Messages Programmatically

A producer client handles serialization, partitioning, batching, and retries on the caller's behalf. When you call send() or produce(), the client serializes the key and value (often with an Avro or JSON serializer tied to a schema registry), applies a partitioner to pick a target partition, and buffers the record into an internal accumulator until a batch is flushed based on linger.ms or batch.size. Delivery is asynchronous by default: the call returns a future or invokes a callback, so production code must check that callback for errors like NotEnoughReplicasException rather than assuming the send succeeded just because the function call returned.

🏏

Cricket analogy: It's like a batsman calling for a quick single — the shot (send call) is played instantly, but the run only counts once the fielder's throw and the umpire's decision (the delivery callback) confirm it, not the moment the bat makes contact.

python

from confluent_kafka import Producer

conf = {
    'bootstrap.servers': 'broker1:9092,broker2:9092',
    'acks': 'all',
    'enable.idempotence': True,
    'linger.ms': 20,
    'batch.size': 65536,
}
producer = Producer(conf)

def delivery_report(err, msg):
    if err is not None:
        print(f'Delivery failed for {msg.key()}: {err}')
    else:
        print(f'Delivered to {msg.topic()} [{msg.partition()}] @ {msg.offset()}')

for order_id in range(1000):
    producer.produce(
        topic='orders',
        key=str(order_id).encode('utf-8'),
        value=f'{{"orderId": {order_id}, "status": "created"}}'.encode('utf-8'),
        callback=delivery_report,
    )
    producer.poll(0)

producer.flush(10)

Consuming Messages Programmatically

A consumer client joins a consumer group, receives a partition assignment from the group coordinator, and repeatedly calls poll() to fetch batches of records. The application is responsible for processing each record and then committing offsets, either automatically on a timer (enable.auto.commit=true, risking reprocessing on crash) or manually after processing completes (commitSync or commitAsync, giving at-least-once guarantees). Long-running processing inside the poll loop without calling poll() again within max.poll.interval.ms causes the coordinator to treat the consumer as dead and trigger a rebalance, so CPU-heavy work is often offloaded to a separate thread pool while the poll loop stays lightweight.

🏏

Cricket analogy: It's like a fielding captain rotating bowlers — if a bowler like Jasprit Bumrah takes too long between overs, the umpire flags a slow over-rate penalty, just as a consumer that stalls past max.poll.interval.ms gets kicked out and triggers a rebalance.

Most clients expose a poll timeout and a max.poll.records (or equivalent) setting. Keep the per-poll processing time well under max.poll.interval.ms — a common pattern is to hand records off to a bounded worker queue and let the poll loop return quickly, so the consumer keeps sending heartbeats to the group coordinator.

Client Configuration and Delivery Guarantees

The delivery guarantee your application actually gets depends on how the producer and consumer are configured together, not on Kafka alone. Setting acks=all with enable.idempotence=true and min.insync.replicas=2 on the producer side prevents duplicate or lost writes from retries, but end-to-end exactly-once still requires either Kafka transactions (initTransactions, beginTransaction, sendOffsetsToTransaction, commitTransaction) or idempotent downstream writes keyed by the message's own identifier. On the consumer side, committing offsets only after the corresponding side effect (a database write, an API call) is durably applied is what actually yields at-least-once processing; committing before processing risks silent data loss if the consumer crashes mid-batch.

🏏

Cricket analogy: It's like DRS review protocol — a decision (write) only stands as final once both the third umpire and on-field umpire (idempotent producer plus min.insync.replicas) independently confirm it, preventing a wrongly given-out batsman from being reinstated later.

Committing consumer offsets before finishing the corresponding processing (a common shortcut with enable.auto.commit=true) can silently drop data: if the process crashes after the commit but before the side effect completes, that message is never retried. Prefer manual commits after processing, or use Kafka transactions to commit offsets and output atomically.

All Kafka client libraries speak the same wire protocol, so producers and consumers in different languages interoperate freely.
Producers batch and serialize records asynchronously; always check the delivery callback rather than assuming a send() call guarantees success.
Consumers must call poll() frequently enough to stay within max.poll.interval.ms or the group coordinator triggers a rebalance.
enable.auto.commit=true risks reprocessing or data loss; manual offset commits after processing give safer at-least-once semantics.
acks=all, enable.idempotence=true, and min.insync.replicas together prevent duplicate or lost writes from producer retries.
True exactly-once semantics require Kafka transactions or idempotent downstream writes, not just producer/consumer config alone.
Offload heavy per-record processing to worker threads so the poll loop stays responsive and heartbeats keep flowing.