Apache Iceberg: The Hub of an Emerging Data Service Ecosystem?
Engineers at Netflix and Apple created Apache Iceberg several years ago to address the performance and usability challenges of using Apache Hive tables in large and demanding data lake environments. Now the data table format is the focus of a burgeoning ecosystem of data services that could automate time-consuming engineering tasks and unleash a new era of big data productivity.
Apache Hive emerged over a decade ago to make Apache Hadoop clusters look and function more like standard relational databases accessible through SQL. While Hadoop usage has waned in the age of cloud data lakes like AWS S3 and Azure Data Lake Storage (ADLS), the Hive legacy continues, both as a query engine for large, batch-oriented analytic jobs but arguably more so as a table format and a metadata catalog used by other query engines, including Apache Spark and Presto, among others.
In this manner, Hive acts as a unifying layer that enables these engines to function on a common set of data stored in Hadoop clusters and S3-compatible data lakes. While Hive marked a significant step forward in big data storage and analytics a decade ago, its technical limitations today are forcing data engineers and analysts to embark upon expensive and time-consuming workarounds to store and analyze massive data sets effectively.
One of the big problems with Hive is that it doesn’t adapt well to changing datasets. It can handle static data just fine, but if a user or an application makes changes to the data, such as an ETL job that updates a Parquet file, then those changes have to be coordinated with other applications or users. If this coordination does not happen, then the data is at risk of becoming corrupt and giving the wrong answer when queried.
This was one of the main drivers behind the Apache Iceberg project. Engineers at Apple and Netflix started the Iceberg project around 2018 to address the limitations in using Hive tables to store and query massive data sets. Ryan Blue, a senior engineer at Netflix and the PMC Chair of the Apache Iceberg project, recently discussed the genesis of Iceberg and the direction it’s headed in a session at the Subsurface 2021 conference, which was sponsored by Dremio and held last month.
“Iceberg exists because Netflix slowly realized we needed a new table format,” Blue said. “Many different services and engines were using Hive tables. But the problem was, we didn’t have that correctness guarantee. We didn’t have atomic transactions. And sometimes changes from one system caused another system to get the wrong data and that sort of issue caused us to just never use these services, not make change to our tables, just to be safe.”
The number one goal of the Iceberg project was to ensure correctness in the data, Blue said.
“Quite simply, tables shouldn’t lie to you when you query them,” he said. “It’s a really simple thing. But we survived for a very, very long period of time where these tables were being updated, or your file system was, say, S3 and didn’t provide a consistent listing, your tables could easily lie to you.”
Iceberg, which is written in Java and also offers a Scala API, effectively solves this dilemma by enforcing transactional consistency in the data, even when it’s accessed by multiple applications. According to Dremio’s description of Iceberg, the Iceberg table format “has similar capabilities and functionality as SQL tables in traditional databases but in a fully open and accessible manner such that multiple engines (Dremio, Spark, etc.) can operate on the same dataset.”
In addition to support for atomic transactions, the second major obstacle the Iceberg project tackled was enabling operations to be performed at a finer-grained level than simply partitioning the data level, Blue said.
“We needed to be able to rewrite data at the file level in order to do more efficient writes,” he said. “We wanted appends that could append to multiple partitions at the time.
