Hadoop Study Notes

Application Master

How YARN's per-application ApplicationMaster negotiates resources, drives task execution, and recovers from failure.

Blocks and Replication

How HDFS splits files into large blocks and replicates them across the cluster to guarantee durability and availability.

Building a Word Count Job

A hands-on walkthrough of writing, compiling, and running the canonical Hadoop MapReduce word count program from mapper to driver.

Hadoop Best Practices

A practical guide to designing, tuning, and operating Hadoop clusters so they stay reliable and performant as data volumes grow.

Hadoop Daemons

What NameNode, DataNode, ResourceManager, NodeManager, and the other Hadoop background processes actually do, and how they interact.

Hadoop Interview Questions

The core HDFS, MapReduce, and YARN questions Hadoop interviewers ask most often, with the reasoning behind strong answers.

Hadoop Quick Reference

A condensed cheat sheet of the HDFS and YARN commands, key configuration properties, and defaults you reach for daily.

Hadoop Security with Kerberos

How Kerberos authentication secures a Hadoop cluster, covering the KDC, tickets, keytabs, SPNEGO, delegation tokens, and how authorization layers like Ranger b…

Hadoop vs Spark

A technical comparison of MapReduce's disk-based batch model and Spark's in-memory DAG execution, and when to reach for each.

HBase Basics

An introduction to Apache HBase's column-oriented data model and its RegionServer, HMaster, and compaction-driven architecture on HDFS.

HDFS Architecture

How HDFS organizes a cluster around a master/worker design so it can store and stream massive files reliably across commodity hardware.

HDFS Commands

The essential hdfs dfs and hdfs dfsadmin commands for navigating, managing, and administering data in HDFS from the command line.

HDFS Read/Write Workflow

Step-by-step walkthrough of what actually happens inside HDFS when a client writes a new file or reads an existing one.

Hive Basics

An introduction to Apache Hive's SQL-on-Hadoop model, covering its architecture, the Metastore, table design, and query execution.

MapReduce Programming Model

A conceptual tour of the map-then-reduce programming model that lets Hadoop process massive datasets by splitting work into independent, parallelizable tasks.

NameNode and DataNodes

A closer look at HDFS's two core daemons: the NameNode that governs metadata and the DataNodes that physically store block data.

Pig Basics

An introduction to Apache Pig's procedural data-flow language, Pig Latin, its execution model, and how it compares to Hive.

ResourceManager and NodeManager

A deep dive into YARN's two master/worker daemons -- the ResourceManager's scheduling and the NodeManager's per-node container enforcement.

Running Jobs on YARN

The end-to-end path of submitting, monitoring, and debugging a job on a YARN-managed Hadoop cluster.

Scheduling in YARN

How YARN's pluggable schedulers -- FIFO, Capacity, and Fair -- decide which application gets the next available container.

Setting Up a Hadoop Cluster

A practical walkthrough of planning and configuring a Hadoop cluster, from single-node pseudo-distributed setups to multi-node production layouts.

Sqoop and Flume

How Sqoop bulk-transfers structured data between RDBMSs and Hadoop, and how Flume ingests continuous streaming log data, covering their core mechanics and conf…

The Hadoop Ecosystem

A tour of the major projects that surround core Hadoop — Hive, Pig, HBase, Sqoop, Flume, ZooKeeper, and Oozie — and how they fit together.

The Map Phase

How Hadoop turns raw input splits into intermediate key-value pairs, including input splitting, mapper lifecycle, local aggregation, and partitioning.