Particularly with the industry spotlight on Snowflake following its recent IPO, there’s no shortage of discussion right now around cloud data warehouses, cloud data lakes, and how the two overlap – or don’t. For many enterprise data and analytics professionals trying to modernize to support ML and AI, there’s still a good deal of confusion on what each type of data solution offers and where the key differences lie.
In this primer, I’ll look at the strengths of each data platform and what each is built to excel at. While cloud Data warehouse and cloud data lakes may solve disparate issues, they can – and executed right, should – complement one another. Used in tandem and backed by the power of the cloud, these two architectures can more fully harness the complete data and analytics picture to deliver the value and business insight that enterprises continue to seek out.
Cloud data warehouses are a decade-old technology that enables analytics by using a mostly relational processing engine – structuring data via tables and columns. Generally categorized as schema-on-write, writes to the Data warehouse must adhere to previously established schema. This is true for any deployment style, including cloud data warehouses.
Naturally, SQL is the universal language of cloud data warehouses. JSON with SQL extensions and similar solutions can also allow for semi-structured data and schema-on-read functionality. However, these solutions add in prohibitively strict ACID transaction overhead. Many non-SQL transactions do not require this: schema-on-read can naturally support these applications, while utilizing less stringent transaction semantics and delivering better performance.
Cloud data warehouses also necessitate data to be cleaned and structured in close alignment with the questions and analysis that business applications are enlisted to solve. Any and all necessary schema changes require a long, intensive, and manual process that includes design work and landing the data in preparation for analysis processes.
Data warehouse relational engines enable advanced analysis by allowing application developers to write user-defined functions (UDFs) and user-defined aggregates (UDAs) – collectively known as user-defined extensions (UDXs). Leveraging UDXs can equip business analytics with a feature set surpassing what can be accomplished using standard SQL. UDXs are used in the same way as other standard SQL functions and aggregates in SQL statements. UDXs offer a full range of use cases and levels of complexity, from simply validating URLs all the way through to statistical functions, encryption/decryption, and compression/decompression.
Cloud data warehouses are commonly tapped to analyze historical data, support business intelligence applications, and fulfill business analysts’ needs for interactive reporting and other ad hoc tasks. For example, a data warehouse might enable a vendor to analyze their product inventory and sales by location, drilling down into data by country, region, and city. The organization can then leverage those insights to better optimize its supply chain and sales processes.
Cloud Data lakes are generalized data processing platforms.
