What is a Heartbeat in Distributed Systems?
Learn what a heartbeat is in distributed systems: liveness signals, timeout tuning, and how failure detection triggers failover.
Expected Interview Answer
A heartbeat in distributed systems is a small periodic signal one node sends to another (or to a coordinator) purely to prove it is still alive, and its absence for a configured timeout is treated as evidence that the node has failed, triggering failover or re-election.
In practice a heartbeat is a lightweight message, often just a ping with a sequence number or timestamp, sent every few seconds over a persistent connection or via a shared registry. A monitoring component or peer tracks the last time it heard from each node, and if that gap exceeds a timeout threshold, the node is marked unhealthy and removed from load balancing, its work is reassigned, or a leader election is triggered in consensus systems like Raft. Heartbeats underpin failure detection in load balancers, cluster managers like Kubernetes, and leader-based systems, but they only give a probabilistic, not perfect, signal of liveness because network partitions can make a healthy node look dead. Designing a heartbeat system means choosing an interval and timeout that balances fast failure detection against false positives from transient network hiccups.
- Enables automatic detection of failed nodes without manual intervention
- Drives failover: removing dead nodes from load balancer pools
- Triggers leader re-election in consensus protocols
- Provides a simple, low-overhead liveness signal across a cluster
AI Mentor Explanation
A heartbeat is like a fielding captain checking in with every fielder over the radio between overs just to confirm they are still in position and fit to play. If a fielder does not respond within a couple of checks, the captain assumes an injury and calls on the substitute, exactly like a coordinator declaring a node dead after missed heartbeats and reassigning its work. This periodic check-in is deliberately lightweight, a quick "you good?" rather than a full fitness test, mirroring how heartbeats are small pings rather than full health reports.
Step-by-Step Explanation
Step 1
Send periodic pings
Each node sends a lightweight signal (timestamp or sequence number) to a peer or coordinator at a fixed interval.
Step 2
Track last-seen time
The receiver records when it last heard from each node it monitors.
Step 3
Detect timeout
If no heartbeat arrives within a configured timeout, the node is marked as suspected failed.
Step 4
Trigger failover
The system removes the node from routing, reassigns its work, or starts a leader election as appropriate.
What Interviewer Expects
- Defining a heartbeat as a lightweight, periodic liveness signal
- Explaining the role of interval and timeout tuning
- Connecting heartbeats to concrete failure-detection use cases (load balancers, leader election)
- Acknowledging heartbeats give a probabilistic, not perfect, liveness signal
Common Mistakes
- Setting the timeout too short, causing false positives during transient network blips
- Setting the timeout too long, delaying failure detection significantly
- Assuming a missed heartbeat always means the node is actually down (network partitions can lie)
- Forgetting heartbeats add ongoing network and processing overhead at cluster scale
Best Answer (HR Friendly)
“A heartbeat is a small, regular signal a node sends just to say "I am still alive." If a system stops receiving those signals from a node within an expected time window, it assumes that node has failed and takes action, like removing it from a load balancer or electing a new leader. It is a simple mechanism, but tuning how often to check and how long to wait is important to catch real failures fast without overreacting to a brief network hiccup.”
Code Example
const HEARTBEAT_INTERVAL_MS = 2000;
const TIMEOUT_MS = 6000; // miss ~3 heartbeats before declaring dead
const lastSeen = new Map();
function recordHeartbeat(nodeId) {
lastSeen.set(nodeId, Date.now());
}
function checkForFailures(onNodeFailed) {
const now = Date.now();
for (const [nodeId, ts] of lastSeen.entries()) {
if (now - ts > TIMEOUT_MS) {
onNodeFailed(nodeId);
lastSeen.delete(nodeId);
}
}
}
setInterval(() => checkForFailures((nodeId) => {
console.log("Node marked failed, reassigning work:", nodeId);
}), HEARTBEAT_INTERVAL_MS);Follow-up Questions
- How do you choose a heartbeat interval and timeout for a latency-sensitive system?
- How can a network partition cause a healthy node to be wrongly marked dead?
- How does heartbeat-based failure detection relate to leader election in Raft?
- How would you reduce heartbeat overhead in a cluster with thousands of nodes?
MCQ Practice
1. What is the primary purpose of a heartbeat in a distributed system?
A heartbeat is a lightweight liveness signal, not a data transfer or backup mechanism.
2. What typically happens when a node misses its heartbeat past the configured timeout?
A missed heartbeat past the timeout triggers failure handling: removal from load balancing, work reassignment, or leader re-election.
3. Why is heartbeat-based failure detection considered probabilistic rather than perfect?
Network issues can prevent a healthy node’s heartbeats from arriving, causing a false failure suspicion.
Flash Cards
What is a heartbeat in distributed systems? — A small periodic signal proving a node is alive; its absence past a timeout implies failure.
What does a missed heartbeat trigger? — Removal from routing, work reassignment, or leader re-election.
Why tune the heartbeat interval carefully? — Too short causes false positives from network blips; too long delays real failure detection.
Is heartbeat failure detection perfectly accurate? — No — it is probabilistic; network partitions can make a healthy node look dead.