--- title: Lakehouse table formats compared url: https://dineshkarthik.me/learn/lakehouse-table-formats type: lesson author: Dineshkarthik Raveendran date_modified: 2026-06-06 description: "An interactive comparison of Apache Iceberg, Delta Lake, Apache Hudi, Apache Paimon, and DuckLake: write strategies, time travel, maintenance and performance, catalogs, streaming and CDC, engine support, the convergence trend, and a pragmatic decision guide." --- # Lakehouse table formats compared Iceberg, Delta, Hudi, Paimon - and now DuckLake - turn files in a lake into real tables, but they were built for different jobs. Here is what actually differs, why the format war is cooling, and how to choose without regret. ## Why table formats exist A data lake is just files in object storage. The old "Hive" approach treated a folder of those files as a table and its sub-folders as partitions. That falls apart the moment you need atomic commits, safe schema changes, row-level updates, or concurrent writers. An **open table format** fixes this by adding a metadata layer that tracks exactly which files make up the table, with transactions and history on top. Four established formats dominate, with an emerging fifth that rethinks where the metadata should live - and none started from the same place: - **Apache Iceberg** - born at Netflix, vendor-neutral, analytics-first, broadest engine support. - **Delta Lake** - born at Databricks (now Linux Foundation), tightest Spark integration. - **Apache Hudi** - born at Uber for upserts, CDC, and incremental processing. - **Apache Paimon** - born from Flink (formerly Flink Table Store), streaming-first. - **DuckLake** - from the DuckDB team (v1.0, April 2026), keeps your data in Parquet but stores all table metadata in a SQL database instead of metadata files. ## The five formats, side by side Explore the comparison below. Click any row to read what the difference means in practice, or focus a single format to see its column clearly. | Dimension | Iceberg | Delta | Hudi | Paimon | DuckLake | | --- | --- | --- | --- | --- | --- | | Write strategy | COW + MOR, v3 deletion vectors | COW + deletion vectors | Native dual COW / MOR | LSM-tree MOR (native) | COW on Parquet; inlined changes | | Row-level deletes | v3 deletion vectors | Deletion vectors | COW & MOR deletes | Changelog deletes | Delete files via catalog | | ACID & concurrency | Snapshot isolation (CAS) | Optimistic concurrency | Non-blocking CC (1.0+) | LSM log + snapshots | SQL DB txns; multi-table | | Time travel | Snapshot / timestamp AS OF | VERSION / TIMESTAMP AS OF | Query by commit / timestamp | Snapshot / tag time travel | AT (VERSION / timestamp) | | Streaming & CDC | Analytics-first; v3 changelog | Streams; Auto CDF + row tracking | Strong CDC & incremental | Streaming-first changelog | Batch-first; snapshot change feed | | Catalogs & governance | REST catalog; Polaris, Unity | Unity; catalog-managed (4.1) | Hive / Glue; Polaris (1.1) | REST catalog (1.3); Hive | Catalog-native (SQL DB) | | Schema & partition evolution | Schema + partition evolution | Schema; limited partition | Schema; limited partition | Schema; manual partition | Schema + evolvable partitions | | Hidden partitioning | Yes (transforms) | No (explicit columns) | No (explicit columns) | Partitioned, no hidden transforms | Transforms + zone-map pruning | | Maintenance operations | rewrite\_data\_files, expire\_snapshots | OPTIMIZE, VACUUM, liquid clustering | Compaction, cleaning, clustering | Automatic LSM compaction | CHECKPOINT: compact, expire, clean | | Performance & data layout | Hidden partitioning + scan planning | Liquid clustering + deletion vectors | Clustering + multi-index + stats V2 | LSM + incremental clustering | SQL-catalog planning + zone maps | | Engine support | Broadest multi-engine | Spark-first, growing | Spark / Flink | Flink-native; others limited | DuckDB-native; growing | | Maturity & momentum | v3 GA cross-vendor | Largest Spark base | Proven at scale | Newer, growing fast | v1.0 (2026), newest | ## How they really differ Strip away the marketing and the differences come down to a few axes. ### Write strategy: copy-on-write vs merge-on-read Copy-on-write rewrites whole data files when rows change - fast reads, heavier writes. Merge-on-read records changes in small delete or log files and merges them at read time - cheap, quick writes at a little read cost, cleaned up by compaction. Hudi and Paimon were designed around merge-on-read; Iceberg and Delta began copy-on-write and have now converged at the data layer - with Iceberg v3 GA on Databricks, Snowflake, and AWS, both share Parquet plus deletion vectors, so a write in one is interchangeable with the other without conversion. **DuckLake** writes plain Parquet with delete files too, but routes every change - even small inlined ones - through its SQL catalog, so commits are near-instant. ### Streaming and CDC This is the sharpest divide. **Paimon** is streaming-first and Flink-native, with first-class changelog producers. **Hudi** has deep CDC and incremental-query support and upsert DNA from its Uber origins. **Iceberg** is analytics-first (it added changelog reads in v3) and **Delta** now ships Auto CDF and row tracking in 4.1, which makes change feeds easier, though it is still not changelog-native the way Paimon and Hudi are. **DuckLake** is batch-first, exposing changes between snapshots rather than a streaming changelog. ### Catalogs, governance, and partitioning The catalog is now where the real contest sits. Both Iceberg and Delta 4.1 support catalog-managed tables, and the live rivalry is Unity Catalog versus Apache Polaris (an Apache top-level project since February 2026). Iceberg still has the richest catalog ecosystem - the REST catalog spec plus open implementations (Polaris, Nessie, Lakekeeper) and managed options (Glue, Unity, Snowflake) - and the most flexible partitioning, with true hidden partitioning and partition-spec evolution. **DuckLake** is catalog-native by design: the SQL database is the catalog, which is what makes its commits and query planning fast. The others largely expose partition columns you manage yourself. ## The format war is cooling The most important development is not a winner - it is **convergence**. At the data layer the fight is effectively over: with Iceberg v3 now GA on Databricks, Snowflake, and AWS, Iceberg and Delta share Parquet, deletion vectors, variant, and row tracking, so you can write in one and read in many without copies: - **Delta UniForm** lets a Delta table emit Iceberg (and Hudi) metadata over the same Parquet files, so other engines can read it without copies. - **Apache XTable** (formerly OneTable, still Apache Incubating) translates metadata between Iceberg, Delta, and Hudi - vendor-neutral, no data rewrite. - **The Iceberg REST catalog** is becoming the de-facto standard for governed, multi-engine access, with Apache Polaris (a top-level project since February 2026) as an open implementation. - **The next step is shared metadata.** Iceberg v4 and Delta 5.0 are being designed around a common adaptive metadata tree (Delta RFC #6640), Hudi 1.1 added a pluggable table-format framework, and DuckLake takes a different route entirely - putting the whole catalog in a SQL database. The honest caveat: convergence is real at the data-and-read layer, but on write you still commit to one format and, increasingly, one catalog. So choose for your write path and your catalog - then let UniForm, XTable, and the REST catalog handle the reads. ## Which should you choose? There is no single best format - only a best fit for your situation. Answer a few questions for a starting point, then read the guide below. What is your primary workload? Batch analytics / BI Streaming / low-latency CDC Frequent upserts on existing data Primary engine or platform? Databricks / Spark-first Flink-first Trino / Dremio / many engines Snowflake / BigQuery What matters most? Vendor-neutral, avoid lock-in Simplicity + tight Spark integration Lowest-latency freshness Best upsert performance Do many engines need to read the same table? Yes - several engines read it Mostly one engine ### A pragmatic guide - **Vendor-neutral analytics, many engines:** Iceberg. Broadest support and an open REST catalog. - **Databricks-centric shop:** Delta as primary, with UniForm for Iceberg reads elsewhere. - **Streaming-first on Flink:** Paimon; on Spark with strong CDC, Hudi. - **Heavy upserts / incremental on Spark:** Hudi. - **DuckDB-centric, simple lakehouse:** DuckLake - metadata in a SQL database, data in Parquet, very few moving parts. - **The catalog now matters more than the format:** at the data layer Iceberg and Delta have converged, so invest in your catalog (Unity Catalog or Apache Polaris) and let interop handle the reads. ## Create the same table in each The same orders table, five formats. Notice how each one's defaults reveal its priorities - Iceberg's hidden partitioning, Hudi's primary key and merge-on-read, Paimon's changelog producer, DuckLake's SQL catalog. ``` -- Apache Iceberg: hidden partitioning, v3 deletion vectors for deletes CREATE TABLE ecom.orders ( order_id BIGINT, customer_id BIGINT, order_date DATE, amount DECIMAL(10,2), status STRING ) USING iceberg PARTITIONED BY (day(order_date), bucket(16, customer_id)) TBLPROPERTIES ('format-version' = '3'); ``` ``` -- Delta Lake 4.1: liquid clustering, deletion vectors, catalog-managed via Unity CREATE TABLE ecom.orders ( order_id BIGINT, customer_id BIGINT, order_date DATE, amount DECIMAL(10,2), status STRING ) USING delta CLUSTER BY (order_date, customer_id) TBLPROPERTIES ( 'delta.enableDeletionVectors' = 'true', 'delta.universalFormat.enabledFormats' = 'iceberg' -- expose Iceberg reads ); ``` ``` -- Apache Hudi: merge-on-read, primary key, ordering field for upserts CREATE TABLE ecom.orders ( order_id BIGINT, customer_id BIGINT, order_date DATE, amount DECIMAL(10,2), status STRING, updated_at TIMESTAMP ) USING hudi PARTITIONED BY (order_date) TBLPROPERTIES ( 'type' = 'mor', 'primaryKey' = 'order_id', 'preCombineField' = 'updated_at', 'hoodie.table.cdc.enabled' = 'true' ); ``` ``` -- Apache Paimon (Flink): streaming primary-key table with a changelog CREATE TABLE ecom.orders ( order_id BIGINT, customer_id BIGINT, order_date STRING, amount DECIMAL(10,2), status STRING, PRIMARY KEY (order_id) NOT ENFORCED ) PARTITIONED BY (order_date) WITH ( 'changelog-producer' = 'lookup', 'bucket' = '16', 'file.format' = 'parquet' ); ``` ``` -- DuckLake: metadata in a SQL catalog (Postgres here), data in Parquet INSTALL ducklake; ATTACH 'ducklake:postgres:dbname=catalog host=localhost' AS lake (DATA_PATH 's3://warehouse/ecom/'); CREATE TABLE lake.orders ( order_id BIGINT, customer_id BIGINT, order_date DATE, amount DECIMAL(10,2), status VARCHAR ); -- Iceberg-style partition transforms, tracked in the catalog ALTER TABLE lake.orders SET PARTITIONED BY (day(order_date), bucket(16, customer_id)); ``` All five keep data in open columnar files (usually Parquet). What changes is where the metadata lives - files for most, a SQL database for DuckLake - and the defaults each format encourages. ## Key takeaways - All five formats keep data in open columnar files (usually Parquet) and differ mainly in metadata design, write strategy, and streaming posture. - Paimon and Hudi are strongest for streaming and CDC; Iceberg and Delta have converged at the data layer with deletion vectors; DuckLake is catalog-native and batch-first. - The catalog layer - Unity Catalog versus Apache Polaris - is now the main governance battleground, not the file format alone. - Choose by workload: analytics breadth (Iceberg), Spark-native base (Delta), upserts and CDC (Hudi), Flink streaming (Paimon), or DuckDB-centric simplicity (DuckLake). - Interoperability is improving through UniForm, Apache XTable, and the Iceberg REST catalog - but you still pick one format on write.