Big Data Architecture in Data Processing and Data Access
Over the past 20+ years, it has been amazing to see how IT has been evolving to handle the ever growing amount of data, via technologies including relational OLTP (Online Transactional Processing) database, data warehouse, ETL (Extraction, Transformation and Loading) and OLAP (Online Analytical Processing) reporting, big data and now AI, Cloud and IoT. All these technologies were enabled by the rapid growth in computational power, particular in terms of processors, memory, storage, and networking speed. The objective of this article is to summarize, first, the underlying principles on how to handle large amounts of data and, second, a thought process that I hope can help you get a deeper understanding of any emerging technologies in the data space and come up with the right architecture when riding on current and future technology waves.
In a data pipeline, data normally go through 2 stages: Data Processing and Data Access. For any type of data, when it enters an organization (in most cases there are multiple data sources), it is most likely either not clean or not in the format that can be reported or analyzed directly by the eventual business users inside or outside of the organization. Data Processing is therefore needed first, which usually includes data cleansing, standardization, transformation and aggregation. The finalized data is then presented in the Data Access layer – ready to be reported and used for analytics in all aspects. Data Processing is sometimes also called Data Preparation, Data Integration or ETL; among these, ETL is probably the most popular name.
Data processing and data access have different goals, and therefore have been achieved by different technologies. Data Processing for big data emphasizes “scaling” from the beginning, meaning that whenever data volume increases, the processing time should still be within the expectation given the available hardware. The overall data processing time can range from minutes to hours to days, depending on the amount of data and the complexity of the logic in the processing. On the other hand, data access emphasizes “fast” response time on the order of seconds. On a high level, the scalability of data processing has been achieved mostly by parallel processing, while fast data access is achieved by optimization of data structure based on access patterns as well as increased amounts of memory available on the servers.
In order to clean, standardize and transform the data from different sources, data processing needs to touch every record in the coming data. Once a record is clean and finalized, the job is done. This is fundamentally different from data access – the latter leads to repetitive retrieval and access of the same information with different users and/or applications. When data volume is small, the speed of data processing is less of a challenge than compared to data access, and therefore usually happens inside the same database where the finalized data reside. As the data volume grows, it was found that data processing has to be handled outside of databases in order to bypass all the overhead and limitations caused by the database system which clearly was not designed for big data processing in the first place. This was when ETL and then Hadoop started to play a critical role in the data warehousing and big data eras respectively.
The challenge of big data processing is that the amount of data to be processed is always at the level of what hard disk can hold but much more than the amount of computing memory that is available at a given time. The fundamental way of efficient data processing is to break data into smaller pieces and process them in parallel.
