Apache Iceberg
Apache Iceberg is an open-source table format for huge analytic datasets that brings database-like features such as schema evolution, ACID transactions, and time travel to data stored in a data lake.
Definition
Apache Iceberg is an open-source table format for huge analytic datasets that brings database-like features such as schema evolution, ACID transactions, and time travel to data stored in a data lake.
Overview
Iceberg was created to solve long-standing problems with using raw files in a data lake as if they were database tables: inconsistent reads during concurrent writes, painful schema changes, and slow query planning caused by having to list huge numbers of files. Rather than being a database itself, Iceberg is a table format — a specification and set of metadata structures that sit on top of file storage (such as Parquet files in cloud object storage) and let multiple query engines treat that storage as a proper, transactional table. Iceberg tracks table metadata, including schema, partitioning, and snapshots of table state, in a layered metadata tree rather than relying on directory listings, which allows engines to plan queries quickly and safely even as the underlying data changes. This design also enables features like atomic commits, hidden partitioning that avoids user error, schema evolution without rewriting data, and time travel queries that read a table as it existed at a previous snapshot. Because Iceberg is an open specification rather than a single vendor's product, it is supported by a wide range of query engines including Apache Spark, Trino, Presto, and various cloud data warehouses, making it a foundational piece of the modern open data lakehouse architecture. It is frequently discussed alongside competing table formats such as Delta Lake and Apache Hudi, all of which address a similar set of problems with somewhat different design tradeoffs.
Key Features
- Open table format specification, not tied to a single engine
- ACID transactions on top of data lake object storage
- Schema evolution without rewriting existing data
- Hidden partitioning that reduces user query errors
- Time travel queries against historical table snapshots
- Fast query planning via layered metadata instead of file listings
- Broad engine support including Spark, Trino, and Presto