The Art of Abstraction: the continuing separation of compute and storage for data analytics

We recently invited 451 Research VP, Matt Aslett to share his thoughts and observations on the practice of separating the storage and computation of analytic data. That is fundamental to Presto, and allows Presto users to scale their analytic platforms so economically. Here’s what Matt had to say about the practice as a trend…

One of the key trends in the data processing and analytics sector in recent years has been the ongoing separation of compute and storage.

Traditionally, business intelligence and analytic projects have been based on the concept of the enterprise data warehouse, which saw compute and storage combined in a monolithic platform to achieve the performance required for high-performance analytics. More recently, the trend has been toward data lakes, but this was similarly based on a monolithic approach of putting all data in a single environment – initially Hadoop – for storage and analysis.

As a growing volume of data originates in the cloud, however, enterprises are increasingly taking an alternative approach of leaving the data where it resides – which increasingly means cloud storage services such as AWS’s Amazon S3, Microsoft’s Azure Data Lake Store or Google Cloud Storage – and spinning up a separate layer of compute engine products and services to analyze the data as and when required.

This approach of separating compute and storage was pioneered by Google with its Dremel research project, which inspired the emergence of distributed SQL query engines that enabled enterprises to bring their existing SQL analysis skills and tools to data stored in Hadoop and cloud storage.

This separation of compute and storage enables what might be considered an abstracted data lake architecture based on a storage layer that makes use of multiple storage engines – including HDFS itself, but also Hadoop Compatible File Storage (HCFS) such as Amazon S3 or Azure Data Lake Store, as well as relational and non-relational databases – and a separate compute layer that takes advantage of multiple distributed SQL query engines (such as Hive, Presto, Spark, Flink or Impala).

Importantly, independent compute engines such as Presto, originally created at Facebook and commercially supported by Starburst Data, can also be used to query data not just in Hadoop and object storage, but also relational and non-relational databases (including NoSQL and Kafka), providing greater flexibility by enabling federated querying of multiple data sources, in multiple locations, at the same time.

In addition to Google, other companies including Facebook, Netflix and AirBnB are enjoying the benefits of this separation of compute and storage.