Charting the data lake: Using the data models with schema-on-read and
‘So what do you mean when you say “data lake?”’ It is always fun to throw this question into a group of data management professionals—I accept that “fun” is relative term here. Despite the term being in general and wide usage for over five years, the level of variety and elasticity around what different people actually mean by that phrase is pretty impressive. However, over time the general trend for data lakes appears to have been to move away from what was initially seen as primarily a collection of Hadoop/HDFS clusters towards a broader set of enterprise data assets requiring a common set of services for access and a common governance and integration fabric. In many cases the data lake can be defined as a super set of repositories of data that includes the traditional data warehouse, complete with traditional relational technology.
One significant example of the different components in this broader data lake, is in terms of different approaches to the data stores within the data lake. There are a set of repositories that are primarily a landing place of data unchanged as it comes from the upstream systems of record. This data is largely unchanged both in terms of the instances of data and unchanged in terms of any schema that may be implied. There is no attempt to enforce any sort of schema as this data is loaded into the data lake, a schema only comes into play when users try to read these stores—hence they are called schema-on-read data stores. This is in contrast to the types of stores more familiar to people coming from the traditional data warehouse world, where typically a lot of effort is expended in setting up a set of data stores that have a consistent and standard schema enforced from the moment they are created, hence the phrase schema-on-write.
There are many possible combinations of these different approaches to the schema for the data repositories across a data lake, one possible usage of these different storage approaches is here.
The landing zone, the area used to initially store the data coming into the data lake, relies on schema-on-read structures to enable the rapid ingestion of incoming data in its native form. Similarly in the data scientist sandbox, the focus is on the rapid collection of data for discovery and analysis, hence the adoption of schema-on-read here as well. Whereas the central analytics area, intended for use by the broader business community with an emphasis on ensuring standardization and consistency, will likely use schema-on-write. In such cases this need for consistency and standardization outweighs the benefits of the ability to rapidly load the data.
A lot has been written elsewhere about the different uses and pros and cons about these two approaches. In summary, schema-on-read allows for the rapid landing of large amounts of data into the data lake but requires extensive tagging of such data to ensure that it is generally usable across the enterprise.
