Why and When to Avoid S3 as a Data Platform for Data Lakes
Data lakes are all the rage these days in large enterprises. A data lake is a single store for both raw copies of source system data and transformed data for use in tasks like reporting, visualization, advanced analytics, and machine learning.
Object stores (like S3) are becoming the platform of choice for data lakes because of two main reasons:
In this blog post, I will dig deeper into some of the advantages of object-stores that are responsible for their popularity as a platform for data lakes. I will also examine some of the often-underestimated challenges that plague the use of object stores for many data lake use cases.
Object stores like S3 provide eleven 9’s of durability (99.999999999%) and four 9’s of availability (99.99%), and they manage to do it on a virtually unlimited scale, at the unbelievably low price of around $23/TB/month. Contrast this with on-prem data warehouse appliances (DWA) that were quite popular just a few years back. DWA cost tens of thousands of dollars per terabyte, excluding enterprise support. Multi-million-dollar contracts for DWAs which supported only a few hundred terabytes were quite common.
When IT leaders mull over data platform choices for their data lakes, object stores’ $23/TB/month price tag is just too good to resist. It makes sense to use the cheapest storage available for the large volumes of data (from hundreds of terabytes to petabytes) that data lakes are expected to hold. Object stores like S3 seem (incorrectly, as we’ll see later in this post) to represent a thousand-fold pricing advantage over the DWA still in use at many large enterprises.
The scale of storage required for a data lake makes it prohibitively expensive to use architectures like DWA in which storage and compute are coupled together in a single package. Decoupling storage and compute allows us to bring the right amount of on-demand compute to bear on the data that needs to be analyzed, at any given time. This significantly reduces the overall cost of data analytics solutions.
All of these advantages are understandably crucial to fueling the popularity of S3 and other object stores as a platform for data lakes. But object stores come with many challenges that don’t get enough attention. This is especially true for RDBMS-sourced and frequently-refreshed (daily/hourly) data, which forms the bulk of high-quality data in an enterprise.
All object stores, including S3, GCS, and Azure Blob Storage, are immutable. This means that files, once written to the object store, can never be edited. Users can only hard delete the old file and create a new one, or logically delete the old file and create a new one (versioning).
When using S3 as a data platform for RDBMS-sourced, frequently-refreshed data, this leads to the creation of an unwieldy number of small files for each table.
As inserts, updates, and deletes pile up over time, trying to derive the current state of the table becomes exponentially more time- and compute-intensive. Most data scientists balk at this complex undertaking and instead request direct access to source systems, defeating the purpose of using a data lake in the first place.
One solution to lift the responsibility of merging changes from end-users is to partition the data and then re-write the partition(s) that were targeted by the latest inserts, updates, and deletes. This eases the burden on end-users somewhat. However, performance problems remain, especially if the table has a large number of columns and only a subset of those columns are needed for the analysis.
The above solution can be improved by using a columnar format like Apache Parquet or Apache ORC. Columnar formats improve performance significantly through better compression of data and limiting I/O to only the columns needed for the analysis. However, reading Parquet files from languages and tools (like Python, R, or Tableau) remains challenging.
