HDFS Read/Write Workflow

Analogy categoryAnalogies

Neither reads nor writes in HDFS ever move file data through the NameNode; it only ever hands out metadata and locations. When a client opens a file, it first asks the NameNode which blocks make up the file and which DataNodes hold each one, then it streams the actual bytes directly to or from those DataNodes over a separate data-transfer protocol. This split keeps the NameNode's workload proportional to the number of metadata operations rather than the volume of data flowing through the cluster, which is exactly why a single NameNode can coordinate a cluster moving many gigabytes per second.

🏏

Cricket analogy: Like a stadium's ticketing office that only tells you your seat number and gate, while you physically walk to the stand yourself, the NameNode only hands out block locations while data flows directly between client and DataNode.

The Write Path

When a client writes a new file, it first asks the NameNode to create a namespace entry, then requests a list of DataNodes to host the first block's replicas; the NameNode returns an ordered pipeline based on the replica placement policy. The client streams data in small packets (typically 64 KB chunks within a larger 512-byte-checksummed unit) to the first DataNode in that pipeline, which forwards each packet to the second, which forwards it to the third, and acknowledgments flow back up the same pipeline in reverse; only once every DataNode in the pipeline has acknowledged a packet does the client consider it durably written.

🏏

Cricket analogy: Like a fielding relay throw from the boundary to the keeper via a cutoff man, HDFS writes stream data through a pipeline of DataNodes, and only once the throw completes the full relay does the run-out (write) count as confirmed.

bash

# Conceptual write sequence (simplified)
# 1. Client -> NameNode: create('/user/data/sales.csv')
# 2. NameNode -> Client: pipeline = [dn1, dn2, dn3]
# 3. Client -> dn1 -> dn2 -> dn3: stream packets
# 4. dn3 -> dn2 -> dn1 -> Client: acknowledgments flow back
# 5. Client -> NameNode: complete() once all blocks acked

hdfs dfs -put -f large_dataset.parquet /user/data/
# Client blocks until the final block's pipeline is fully acknowledged

The Read Path

Reads follow a mirror-image but simpler flow: the client asks the NameNode for the block locations of a file, and the NameNode returns them sorted by proximity, same node first, then same rack, then off-rack, so the client naturally prefers the closest available replica. The client then opens a direct connection to the chosen DataNode and streams the block, verifying each chunk's checksum as it arrives; if a DataNode is unreachable or a checksum fails, the client transparently falls back to the next DataNode in the list without the application ever seeing an error.

🏏

Cricket analogy: Like a broadcaster always cutting to the nearest camera angle first for the fastest replay, HDFS reads prefer the closest DataNode replica, falling back to a farther one only if the nearest feed drops out.

The client library, DFSClient, is what performs all this pipeline and fallback logic transparently; application code calling the Hadoop FileSystem API (or hdfs dfs) never sees the individual DataNode connections, retries, or checksum verification, it just sees a standard read or write stream succeed or throw an IOException after all retries are exhausted.

Handling Failures Mid-Transfer

If a DataNode in a write pipeline fails mid-transfer, HDFS doesn't abort the write: it removes the failed node from the pipeline, asks the NameNode for a replacement DataNode, and continues streaming to the remaining nodes, with the block later re-replicated to restore the full replication factor once the write completes. This means a client's write only fails outright if every DataNode assigned to a block becomes unreachable simultaneously, which is rare enough that HDFS writes are considered reliable even on a cluster experiencing routine hardware churn.

🏏

Cricket analogy: Like a bowling attack continuing the innings with a replacement bowler when one pulls a hamstring mid-over, HDFS keeps a write going by swapping in a new DataNode rather than abandoning the whole innings.

Because HDFS pipelines require acknowledgment from every DataNode before a write is confirmed, write latency is bounded by the slowest node in the pipeline. A single consistently slow or overloaded DataNode can noticeably degrade write throughput cluster-wide, which is why monitoring per-DataNode latency, not just aggregate cluster throughput, matters for diagnosing performance issues.

The NameNode only ever hands out metadata; actual file bytes flow directly between clients and DataNodes.
Writes stream through a pipeline of DataNodes, with acknowledgments flowing back before data is considered durable.
Reads prefer the closest replica (same node, then same rack, then off-rack) and verify checksums per chunk.
The DFSClient library transparently handles pipeline construction, failover, and checksum verification.
A failed DataNode mid-write is replaced in the pipeline rather than aborting the write outright.
Blocks under-replicated after a mid-write failure are re-replicated automatically afterward.
Write latency is bounded by the slowest DataNode in the pipeline, making per-node monitoring important.

Practice what you learned

1. Does file data ever pass through the NameNode during a read or write?

2. In the HDFS write path, when is a packet considered durably written?

3. How does the NameNode order the DataNode list it returns for a read request?

4. What happens if a DataNode in an active write pipeline fails?

5. What determines the write throughput of an HDFS pipeline?

Was this page helpful?

HDFS Commands

Related Study Notes in Programming

Browse all study notes

Programming

Apache Spark Study Notes

Programming · 30 topics

Programming

Apache Flink Study Notes

Programming · 30 topics

Programming

Snowflake Study Notes

Programming · 30 topics

Programming

Apache Airflow Study Notes

Programming · 30 topics

Programming

dbt (Data Build Tool) Study Notes

Programming · 30 topics

Programming

PowerShell Study Notes

Programming · 30 topics

Topics covered

#Programming#HadoopStudyNotes#HDFSReadWriteWorkflow#HDFS#Read#Write#Workflow#StudyNotes#SkillVeris#ExamPrep

HDFS Read/Write Workflow