Building a Task Queue with RabbitMQ

Analogy categoryAnalogies

Designing a Task Queue: The Basic Shape

A task queue offloads slow or resource-intensive work — sending emails, resizing images, generating PDF reports — from a request-handling process to a pool of background workers. In RabbitMQ this maps naturally onto a durable queue that producers publish task messages to and multiple worker processes consume from; RabbitMQ's default round-robin dispatch spreads tasks across available workers, and because only one worker will ever receive a given message, tasks are naturally load-balanced without any coordination logic in the application.

🏏

Cricket analogy: A task queue distributing image-resize jobs across workers is like a fielding captain rotating bowlers through an over-by-over spell so no single bowler is overworked while the required intensity keeps up.

Reliability: Durability, Acknowledgments, and Prefetch

To survive a broker restart, both the queue and the messages published to it must be marked durable, otherwise everything is lost on restart. On the consumer side, manual acknowledgments (rather than auto-ack) ensure a message is only removed from the queue after the worker confirms it finished the work; if the worker crashes mid-task, RabbitMQ requeues the unacknowledged message for another worker to pick up. The prefetch count (basic_qos) controls how many unacknowledged messages a worker can hold at once — setting it to 1 ensures a slow worker isn't handed a large batch of work while faster workers sit idle.

🏏

Cricket analogy: Setting prefetch to 1 so a slow worker isn't loaded with a backlog is like a captain not stacking all the tail-end batting into one over — you distribute risk based on current form, not a fixed rotation.

Handling Failures: Retries and Dead-Lettering

Tasks fail — a third-party API times out, a malformed payload arrives, a worker crashes. A dead-letter exchange (DLX) gives failed messages somewhere to go instead of looping forever: when a message is rejected (basic_nack with requeue=False) or expires via TTL, RabbitMQ routes it to the configured DLX, from which it can land in a dedicated retry or 'parking lot' queue for inspection. A common pattern combines a per-message retry count carried in a custom header with a short-TTL retry queue that dead-letters back to the main queue after a delay, giving an exponential-backoff-like retry mechanism without external tooling.

🏏

Cricket analogy: A DLX catching a failed run-out review and sending it to the third umpire for a second look is like RabbitMQ routing a rejected message somewhere for further handling rather than discarding it silently.

python

import pika, json

connection = pika.BlockingConnection(pika.ConnectionParameters('localhost'))
channel = connection.channel()

channel.exchange_declare(exchange='tasks.dlx', exchange_type='direct')
channel.queue_declare(queue='tasks.failed', durable=True)
channel.queue_bind(queue='tasks.failed', exchange='tasks.dlx', routing_key='resize_image')

channel.queue_declare(
    queue='resize_image',
    durable=True,
    arguments={'x-dead-letter-exchange': 'tasks.dlx'},
)

channel.basic_qos(prefetch_count=1)

def callback(ch, method, properties, body):
    task = json.loads(body)
    try:
        resize_image(task['image_id'])
        ch.basic_ack(delivery_tag=method.delivery_tag)
    except Exception:
        ch.basic_nack(delivery_tag=method.delivery_tag, requeue=False)

channel.basic_consume(queue='resize_image', on_message_callback=callback)
channel.start_consuming()

Publish tasks with delivery_mode=2 (persistent) to a durable queue, and always pair manual acknowledgment with basic_qos(prefetch_count=1) for a fair, crash-resistant task queue — the two settings together are what make the queue actually reliable, not just functional.

If you use basic_nack with requeue=True on a message that fails deterministically (a malformed payload, for example), you create an infinite retry loop that hammers the worker and the broker; always route deterministic failures to a dead-letter queue instead of requeuing them blindly.

Mark both the queue and messages as durable/persistent to survive broker restarts.
Use manual acknowledgment so a message is only removed after the worker confirms the task succeeded.
Set prefetch count (basic_qos) to a small number, often 1, to balance load fairly across workers of varying speed.
RabbitMQ's default round-robin dispatch naturally load-balances tasks across a worker pool with no extra coordination code.
Use a dead-letter exchange to capture rejected or expired messages instead of losing them or looping forever.
Combine a retry-count header with a TTL-based retry queue to implement backoff-style retries without external tooling.
Never blindly requeue a message that fails deterministically — route it to a dead-letter queue to avoid infinite retry loops.