Apache Iceberg
Apache Iceberg is an open-source table format for analytical data lakes, initially developed by Netflix to solve consistency and scalability problems at petabyte scale.
Publisher review
Apache Iceberg is an open-source table format for analytical data lakes, initially developed by Netflix to solve consistency and scalability problems at petabyte scale. Donated to the Apache Foundation in 2018 and graduated as a top-level project in 2020, Iceberg brings ACID guarantees, time-travel capability, and schema evolution to data stored in object storage (S3, GCS, ADLS) without rewrites. Unlike Hive's explicit directory partitioning, Iceberg manages partitions as metadata, enabling partition evolution and automatic pruning.
Rows can be updated or deleted with SQL semantics via deletion vectors or eager rewrites. The format is genuinely multi-engine compatible—Spark, Flink, Presto, Trino, Hive, and Impala can all read and write the same tables safely. Metadata lives as files in cloud storage, removing the need for external metastores.
Real-time ingestion isn't Iceberg's strength (Adobe hit a 15 transactions/minute ceiling), but streaming patterns like Kafka → Flink → Iceberg work well for micro-batch analytics. Schema evolution without data rewrites remains its primary advantage, especially for teams managing 100 TB–1 PB tables. Operational costs are real: metadata cleanup, compaction jobs, and lock management add infrastructure burden. Iceberg is production-grade and widely adopted (Netflix, Airbnb, ByteDance, Apple, LinkedIn), though smaller teams often find the operational overhead unjustified and migrate to simpler formats.
How it works
-
Schema Evolution Without Rewrites
Add, drop, rename, or reorder columns without reprocessing existing data—metadata operation only.
-
Hidden Partitioning with Partition Evolution
Partitions managed as metadata, not directory structure. Change partition strategy dynamically without rewriting tables.
-
Time Travel and Point-in-Time Queries
Query specific versions of a table by timestamp or snapshot ID; roll back atomically to previous states.
-
Multi-Engine Compatibility
Spark, Flink, Presto, Trino, Hive, Impala, and others read/write the same table format safely with concurrent access.
-
Row-Level Updates and Deletes
MERGE INTO, UPDATE, DELETE SQL commands with ACID semantics; optional deletion vectors defer cleanup cost.
-
Incremental Processing and CDC
Change Data Capture via Flink/Kafka; process only modified data since last snapshot for efficient streaming pipelines.
-
Metadata in Object Storage
No external metastore required—metadata and data both live in S3/GCS/ADLS, enabling cloud-native operations.
Strengths and trade-offs
Strengths
- Schema evolution without ETL rewrites is rare at scale; true competitive advantage for teams managing evolving data models.
- Hidden partitioning eliminates partition-pruning bugs and manual partition maintenance, reducing data engineering friction.
- Genuine multi-engine compatibility—not Spark-centric like Delta or Hudi—enables teams to query with Presto, Trino, Flink simultaneously.
Trade-offs
- Metadata maintenance is operationally expensive: small files, expired snapshots, and orphan data accumulate and require separate compaction jobs that themselves become bottlenecks.
- Real-time write throughput is fundamentally limited (Adobe hit 15 transactions/minute ceiling); unsuitable for high-velocity OLTP or event streams requiring per-second ingestion.
- Requires distributed lock management (DynamoDB, JDBC) for safe concurrent writes; misconfiguration is a common source of metadata corruption in production.
Pricing context
Apache Iceberg is open-source under the Apache 2.0 license at no cost. However, operational expenses are significant: metadata compaction, cleanup jobs, and lock management must be run separately on Spark, Athena, or other compute platforms, which are billable. Managed services (Databricks, Snowflake Polaris, AWS Glue) charge separately for compute, storage, and ancillary operations. Tabular, the company founded by Iceberg's original creators to offer managed hosting, was acquired by Databricks in June 2024 and consolidated into Databricks' product line.
Alternatives
User reviews
No user reviews yet. Be the first to write one.
Sources
Reporting on this tool draws on these publicly available sources.
- iceberg.apache.org — Official Apache Iceberg documentation, version history (1.11.0 as of May 2026), project governance as top-level Apache project
- github.com — Source repository, commit history since inception (2017), maintainer list, active issue tracker, community activity signals
- aws.amazon.com — AWS-independent overview of Iceberg capabilities, use cases, and positioning against Delta Lake and Hudi
- www.dremio.com — Technical deep-dive on schema evolution, hidden partitioning, partition evolution mechanics, and real-world query performance patterns
- www.databricks.com — Databricks acquisition of Tabular (June 2024) and consolidation of Iceberg roadmap; Databricks commitment to Iceberg alongside Delta Lake
- amdatalakehouse.substack.com — 2026 ecosystem survey data (table distribution by size, engine adoption), community sentiment, emerging Iceberg Summit 2025 trends