A Data Engineer’s Guide To Non-Traditional Data Storages
With the rise of and data science, many engineering roles are being challenged and expanded. One new-age role is .
Originally, the purpose of data engineering was the loading of external data sources and the designing of databases (designing and developing pipelines to collect, manipulate, store, and analyze data).
It has since grown to support the volume and complexity of big data. So data engineering now encapsulates a wide range of skills, from web-crawling, data cleansing, distributed computing, and data storage and retrieval.
For data engineering and data engineers, data storage and retrieval is the critical component of the pipeline together with how the data can be used and analyzed.
In recent times, many new and different data storage technologies have emerged. However, which one is best suited and has the most appropriate features for data engineering?
Most engineers are familiar with SQL databases, such as PostgreSQL, MSSQL, and MySQL, which are structured in relational data tables with row-oriented storage.
Given how ubiquitous these databases are, we won’t discuss them today. Instead, we explore three types of alternative data storages that are growing in popularity and that have introduced different approaches to dealing with data.
Within the context of data engineering, these technologies are search engines, document stores, and columnar stores.
In this article, we explore all three technologies: as a search engine, as a document store, and as a columnar store.
By understanding alternative data storage, we can choose the most suitable one for each situation.
To compare these technologies, we’ll examine how they index, shard, and aggregate data.
Each data indexing strategy improves certain queries while hindering others.
Knowing which queries are used most often can influence which data store to adopt.
Sharding, a methodology by which databases divide its data into chunks, determines how the infrastructure will grow as more data is ingested.
Choosing one that matches our growth plan and budget is critical.
Finally, these technologies each aggregate its data very differently.
When we are dealing with gigabytes and terabytes of data, the wrong aggregation strategy can limit the types and performances of reports we can generate.
As data engineers, we must consider all three aspects when evaluating different data storages.
Elasticsearch quickly gained popularity among its peers for its scalability and ease of integration. Built on top of , it offers a powerful, out-of-the-box text search and indexing functionality. Aside from the traditional search engine tasks, text search, and exact value queries, Elasticsearch also offers layered aggregation capabilities.
At this point, MongoDB can be considered the go-to NoSQL database. Its ease of use and flexibility quickly earned its popularity. MongoDB supports rich and adaptable querying for digging into complex documents. Often-queried fields can be sped up through indexing, and when aggregating a large chunk of data, MongoDB offers a multi-stage pipeline.
Alongside the growth of NoSQL’s popularity, columnar databases have also gathered attention, especially for data analytics. By storing data in columns instead of the usual rows, aggregation operations can be executed directly from the disk, greatly increasing performance. A few years ago, Amazon rolled out its hosted service for a columnar store called Redshift.
In many ways, search engines are data stores that specialize in indexing texts.
While other data stores create indices based on the exact values of the field, search engines allow retrieval with only a fragment of the (usually text) field.
By default, this retrieval is done automatically for every field through analyzers.
An analyzer is a module that creates multiple index keys by evaluating the field values and breaking them down into smaller values.
