State Backends

Analogy categoryAnalogies

What a State Backend Controls

A state backend determines two separate things in Flink: where working state lives while the job is running (on-heap Java objects vs. serialized bytes on local disk), and how that state is persisted into a checkpoint or savepoint (typically a distributed filesystem like S3 or HDFS). Since Flink 1.13, the backend is decoupled from the checkpoint storage location, but the runtime storage choice — HashMapStateBackend versus EmbeddedRocksDBStateBackend — still dictates the latency, memory footprint, and maximum state size your job can sustain.

🏏

Cricket analogy: A team's dressing room can keep bats and pads either in an open kit bag everyone can grab instantly (in-memory) or in a locked storage truck parked outside that takes a minute to fetch from (disk-backed) — the tradeoff HashMapStateBackend versus RocksDB makes.

HashMapStateBackend: In-Memory Objects

HashMapStateBackend keeps all working state as regular Java/Scala objects on the JVM heap, which makes reads and writes extremely fast since there's no serialization on every access — only during checkpointing, when state is serialized and written to the configured checkpoint storage (JobManagerCheckpointStorage for tiny state, or FileSystemCheckpointStorage for anything production-sized). Its major constraint is that total state size is bounded by available heap memory and subject to garbage collection pauses, which makes it the right default for jobs with modest state — counts, small windows, deduplication over bounded key spaces — but a poor fit for state that can grow into the tens or hundreds of gigabytes per task manager.

🏏

Cricket analogy: Keeping the entire live scorecard chalked up on a small pavilion board works great for a quick T20 but becomes unreadable clutter if you tried to fit an entire five-day Test's ball-by-ball log on it, just as HashMapStateBackend suits small state but not massive state.

java

StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
env.setStateBackend(new HashMapStateBackend());
env.getCheckpointConfig().setCheckpointStorage("s3://flink-checkpoints/my-job/");
env.enableCheckpointing(30000); // checkpoint every 30 seconds

EmbeddedRocksDBStateBackend: Disk-Backed State

EmbeddedRocksDBStateBackend stores state as serialized byte arrays in an embedded RocksDB instance on local disk (SSD strongly recommended), which means every state access pays a serialization/deserialization cost but total state size is bounded by local disk rather than heap — routinely scaling to hundreds of gigabytes or more per task manager. It also supports incremental checkpointing: instead of re-uploading the full state snapshot every time, RocksDB's LSM-tree structure lets Flink upload only the SST files that changed since the last checkpoint, dramatically cutting checkpoint duration and network cost for large state jobs.

🏏

Cricket analogy: A stadium's archival library stores decades of full match footage on tape rather than in the commentary box's small on-air memory, and only newly recorded matches get added to the archive each week rather than re-copying the whole library, like incremental checkpointing.

RocksDB access always goes through JNI and serialization, so per-record state access is meaningfully slower than HashMapStateBackend — benchmark before assuming RocksDB is a safe universal default; for small-state, latency-sensitive jobs, HashMapStateBackend is usually the better choice.

Choosing Between Them

The practical decision hinges on expected state size and latency sensitivity: if state per task manager comfortably fits in heap with room for GC headroom, HashMapStateBackend gives lower latency and simpler tuning. If state is large — think session windows over months of user history, or joins against slowly-changing large-cardinality dimension tables — EmbeddedRocksDBStateBackend with incremental checkpointing is usually the only viable option, and it's tunable further through RocksDB column family options like block cache size and write buffer size for workloads with specific access patterns.

🏏

Cricket analogy: A club captain picking a squad for a friendly Sunday match doesn't need the elaborate logistics of an international tour, but a national selector planning a five-nation World Cup campaign absolutely does — the scale of the task dictates the tooling, just as state size dictates the backend.

Both backends can be configured either in flink-conf.yaml via state.backend.type: hashmap or rocksdb, or programmatically per job via env.setStateBackend(), and the backend choice can be changed across a savepoint restore since savepoints store state in a backend-independent canonical format.

A state backend controls both runtime state storage (heap vs. local disk) and interacts with checkpoint storage for persistence.
HashMapStateBackend stores state as JVM objects on-heap, giving the fastest access but bounding state size to available memory.
EmbeddedRocksDBStateBackend serializes state to an embedded RocksDB instance on local disk, supporting state sizes far beyond heap capacity.
RocksDB access incurs serialization and JNI overhead on every read/write, making it slower per-access than HashMapStateBackend.
RocksDB supports incremental checkpoints, uploading only changed SST files rather than a full snapshot each time.
Since Flink 1.13, state backend and checkpoint storage are configured independently.
Savepoints use a canonical, backend-independent format, so you can switch backends across a savepoint restore.