Apache Iceberg

Apache Iceberg is an open-source table format for analytical data lakes, initially developed by Netflix to solve consistency and scalability problems at petabyte scale.

Reviewed by 7wData

On this page

Publisher review

Apache Iceberg is an open-source table format for analytical data lakes, initially developed by Netflix to solve consistency and scalability problems at petabyte scale. Donated to the Apache Foundation in 2018 and graduated as a top-level project in 2020, Iceberg brings ACID guarantees, time-travel capability, and schema evolution to data stored in object storage (S3, GCS, ADLS) without rewrites. Unlike Hive's explicit directory partitioning, Iceberg manages partitions as metadata, enabling partition evolution and automatic pruning.

Rows can be updated or deleted with SQL semantics via deletion vectors or eager rewrites. The format is genuinely multi-engine compatible—Spark, Flink, Presto, Trino, Hive, and Impala can all read and write the same tables safely. Metadata lives as files in cloud storage, removing the need for external metastores.

Real-time ingestion isn't Iceberg's strength (Adobe hit a 15 transactions/minute ceiling), but streaming patterns like Kafka → Flink → Iceberg work well for micro-batch analytics. Schema evolution without data rewrites remains its primary advantage, especially for teams managing 100 TB–1 PB tables. Operational costs are real: metadata cleanup, compaction jobs, and lock management add infrastructure burden. Iceberg is production-grade and widely adopted (Netflix, Airbnb, ByteDance, Apple, LinkedIn), though smaller teams often find the operational overhead unjustified and migrate to simpler formats.

Get the AI & data signal, daily.

48k+ subscribers read this every morning. One email, both newsletters. Unsubscribe anytime.

How it works

  1. Schema Evolution Without Rewrites

    Add, drop, rename, or reorder columns without reprocessing existing data—metadata operation only.

  2. Hidden Partitioning with Partition Evolution

    Partitions managed as metadata, not directory structure. Change partition strategy dynamically without rewriting tables.

  3. Time Travel and Point-in-Time Queries

    Query specific versions of a table by timestamp or snapshot ID; roll back atomically to previous states.

  4. Multi-Engine Compatibility

    Spark, Flink, Presto, Trino, Hive, Impala, and others read/write the same table format safely with concurrent access.

  5. Row-Level Updates and Deletes

    MERGE INTO, UPDATE, DELETE SQL commands with ACID semantics; optional deletion vectors defer cleanup cost.

  6. Incremental Processing and CDC

    Change Data Capture via Flink/Kafka; process only modified data since last snapshot for efficient streaming pipelines.

  7. Metadata in Object Storage

    No external metastore required—metadata and data both live in S3/GCS/ADLS, enabling cloud-native operations.

Strengths and trade-offs

Strengths

  • Schema evolution without ETL rewrites is rare at scale; true competitive advantage for teams managing evolving data models.
  • Hidden partitioning eliminates partition-pruning bugs and manual partition maintenance, reducing data engineering friction.
  • Genuine multi-engine compatibility—not Spark-centric like Delta or Hudi—enables teams to query with Presto, Trino, Flink simultaneously.

Trade-offs

  • Metadata maintenance is operationally expensive: small files, expired snapshots, and orphan data accumulate and require separate compaction jobs that themselves become bottlenecks.
  • Real-time write throughput is fundamentally limited (Adobe hit 15 transactions/minute ceiling); unsuitable for high-velocity OLTP or event streams requiring per-second ingestion.
  • Requires distributed lock management (DynamoDB, JDBC) for safe concurrent writes; misconfiguration is a common source of metadata corruption in production.

Pricing context

Apache Iceberg is open-source under the Apache 2.0 license at no cost. However, operational expenses are significant: metadata compaction, cleanup jobs, and lock management must be run separately on Spark, Athena, or other compute platforms, which are billable. Managed services (Databricks, Snowflake Polaris, AWS Glue) charge separately for compute, storage, and ancillary operations. Tabular, the company founded by Iceberg's original creators to offer managed hosting, was acquired by Databricks in June 2024 and consolidated into Databricks' product line.

Alternatives

User reviews

No user reviews yet. Be the first to write one.

Sources

Reporting on this tool draws on these publicly available sources.

  1. iceberg.apache.org — Official Apache Iceberg documentation, version history (1.11.0 as of May 2026), project governance as top-level Apache project
  2. github.com — Source repository, commit history since inception (2017), maintainer list, active issue tracker, community activity signals
  3. aws.amazon.com — AWS-independent overview of Iceberg capabilities, use cases, and positioning against Delta Lake and Hudi
  4. www.dremio.com — Technical deep-dive on schema evolution, hidden partitioning, partition evolution mechanics, and real-world query performance patterns
  5. www.databricks.com — Databricks acquisition of Tabular (June 2024) and consolidation of Iceberg roadmap; Databricks commitment to Iceberg alongside Delta Lake
  6. amdatalakehouse.substack.com — 2026 ecosystem survey data (table distribution by size, engine adoption), community sentiment, emerging Iceberg Summit 2025 trends