What is the Thundering Herd Problem?
Understand the thundering herd problem, why synchronized cache expiry overwhelms databases, and mitigations like coalescing and TTL jitter.
Expected Interview Answer
The thundering herd problem occurs when a large number of processes or requests are all woken up or triggered simultaneously by the same event — such as a popular cache key expiring — and rush to hit a shared resource like a database at once, overwhelming it even though only one of them actually needed to do the work.
The classic cache example: a hot key with a TTL expires, and hundreds of concurrent requests all miss the cache in the same instant, each independently querying the database to regenerate the same value, which can spike database load enough to cause cascading failures. The same pattern shows up in other contexts too — many worker processes woken from a blocking accept() call on a shared socket when only one connection is pending, or many clients retrying at the exact same moment after a shared timeout. Mitigations include request coalescing (only the first miss triggers a database fetch while others wait for and reuse that result), locking around cache regeneration, staggering TTLs with jitter so keys do not all expire simultaneously, and probabilistic early expiration where a cache proactively refreshes a key slightly before it actually expires. The core fix in every case is ensuring that a shared trigger event causes only one unit of work, not N duplicate ones.
- Understanding it prevents cascading database overload from synchronized cache expiry
- Request coalescing avoids duplicate work for the same cache-miss key
- TTL jitter spreads out expirations to avoid synchronized stampedes
- Recognizing the pattern generalizes beyond caching to locks, sockets, and retries
AI Mentor Explanation
The thundering herd problem is like every fan in the stadium standing up and rushing the same single snack stall the instant halftime is announced, even though only a handful actually need refreshments right now. The stall (the database) gets overwhelmed by the synchronized rush, not because demand was truly that high, but because everyone reacted to the exact same trigger at once. A smarter approach staggers breaks by seating section, or has one usher fetch snacks for a whole row instead of each fan queuing individually. That single-trigger, everyone-rushes-at-once failure is exactly the thundering herd problem.
Step-by-Step Explanation
Step 1
Shared trigger event fires
A hot cache key expires, a lock is released, or a scheduled timer fires identically for many clients at once.
Step 2
Many requests wake simultaneously
All clients or processes waiting on that shared event become active at the exact same moment.
Step 3
All rush the same downstream resource
Each one independently queries the database or backend service to redo the same work, multiplying load N times over.
Step 4
Resource is overwhelmed or cascades
The shared resource, sized for normal load, is hit by a synchronized spike and can slow down or fail, sometimes cascading further.
What Interviewer Expects
- Clear definition: many requests triggered by the same event hitting a shared resource simultaneously
- Uses the hot cache key expiry example concretely
- Names at least two mitigations: request coalescing/locking, TTL jitter, probabilistic early expiration
- Recognizes the pattern is not limited to caching (locks, sockets, retries)
Common Mistakes
- Describing it only as “too much traffic” without the synchronized-trigger root cause
- Not knowing any concrete mitigation beyond “add more servers”
- Confusing it with a DDoS attack (thundering herd is an emergent internal pattern, not malicious traffic)
- Forgetting that naive retry-on-failure logic without jitter can itself cause a thundering herd
Best Answer (HR Friendly)
“The thundering herd problem happens when one shared event, like a popular cache entry expiring, causes a huge number of requests to all hit the same backend system at exactly the same moment, even though only one of them really needed to do the work. It overwhelms the system with a synchronized spike, and the fix is usually to make sure only one request actually does the work while the rest wait for and reuse that result, or to spread out the trigger so it does not happen all at once.”
Code Example
const inFlightRequests = new Map()
async function getWithCoalescing(key, fetchFn) {
const cached = await cache.get(key)
if (cached !== null) return cached
// If another request is already fetching this key, wait for it
// instead of triggering a duplicate database query.
if (inFlightRequests.has(key)) {
return inFlightRequests.get(key)
}
const promise = (async () => {
try {
const fresh = await fetchFn()
// jittered TTL avoids many keys expiring at exactly the same time
const jitterSeconds = Math.floor(Math.random() * 30)
await cache.set(key, fresh, { ttlSeconds: 300 + jitterSeconds })
return fresh
} finally {
inFlightRequests.delete(key)
}
})()
inFlightRequests.set(key, promise)
return promise
}Follow-up Questions
- How does request coalescing prevent duplicate database queries on a cache miss?
- What is TTL jitter and how does it reduce the chance of a thundering herd?
- How does probabilistic early expiration differ from a hard TTL cutoff?
- How does the thundering herd problem show up outside of caching, such as with locks or sockets?
MCQ Practice
1. What triggers a classic cache-related thundering herd?
When a popular key expires, all concurrent requests miss simultaneously and independently hit the database for the same value.
2. What does request coalescing do to prevent a thundering herd?
Coalescing ensures a single in-flight fetch serves all concurrent requesters for the same key instead of each triggering a duplicate query.
3. How does TTL jitter help prevent a thundering herd?
Adding a random offset to TTLs spreads out expirations over time instead of many keys expiring simultaneously.
Flash Cards
What is the thundering herd problem? — Many requests triggered by the same event all hit a shared resource simultaneously, overwhelming it.
Classic cache example? — A hot key expires and many concurrent requests all miss and query the database at once for the same value.
Request coalescing? — Only the first miss triggers a fetch; other concurrent requests wait for and reuse that same result.
TTL jitter? — Randomizing expiration times so cache keys do not all expire at the same instant.