What Is the Next Stop For Big Data? Hybrid Service/Analytics Processing (HSAP)

Due to different emphases, traditional databases can be divided into a transaction-focused OLTP system and an analytics-focused OLAP system. With the development of the Internet, the amount of data has increased exponentially, and the single-machine database can no longer meet the needs of the business. Especially in the field of analytics, a query may need to process a large part or even the full amount of data, and the pressure brought by massive data becomes particularly urgent. This contributed to the big data revolution that started with Hadoop technology in the past decade or so and solved the demand for massive data analytics. At the same time, several distributed database products have emerged in the database field to cope with the growth of OLTP scenario data.
To analyze the data in the OLTP system, the standard practice is to synchronize the data in the OLTP system to an OLAP system regularly (e.g. every day). This architecture ensures that analytical queries will not affect online transactions. However, regular synchronization has led to the analytics results not being based on the latest data, and this delay has deprived us of the opportunity to make more timely business decisions. To solve this problem, HTAP architecture has emerged in recent years, which allows us to directly analyze the data in the OLTP database, thus ensuring the timeliness of analytics. Analytics is no longer a unique capability of traditional OLAP systems or big data systems. A natural question is: Since HTAP has the capability of analytics, will it replace big data systems? What is the next stop for big data?
To answer this question, we will take the recommendation system as an example to analyze the typical scenarios of big data systems.
When you see the shopping application showing you what you just want to buy and the short video application playing your favorite music, the recommendation system is playing its magical role. The core goal of an advanced recommendation system is to make personalized recommendations according to the real-time behavior of users. Each interaction between users and the system will optimize the next experience in real-time. To support such a system, the big data technology stack has evolved into a very complex and fragmented system.
The following figure shows a big data technology stack that supports a real-time recommendation system.
To provide a high-quality real-time personalized recommendation, the recommendation system relies heavily on real-time features and continuous updating of models.
Real-time features can be divided into two categories:
These data are also used to generate real-time and offline machine learning samples, and the trained models will be continuously updated to the recommendation system after verification.
What is explained above is the core part of an advanced recommendation system, but this is only the tip of the iceberg of the whole system? Also, a complete set of systems such as real-time model monitoring, verification, analytics, and tuning are needed, which include: using a real-time large screen to view the results of A/B test (3), using interactive analytics (4) for BI, and refining and tuning the model. Also, the operation will use various complicated queries to gain insight into the progress of the business and carry out targeted marketing utilizing customer-targeting and product recommendations.
This example shows a very complex but typical big data scenario, from real-time data import (a) to pre-aggregation (b), from data service (1), continuous aggregation (3), to interactive query (4), to batch processing (2). Such complex scenarios have very diversified requirements for big data systems. We have seen two new trends in the practice of building these systems.
Real-time: Business needs to quickly gain business insight from the data just collected. The written data needs to be visible in seconds or even sub-seconds.


