Data Lake: A more Technical Point of View
Companies have come to realize of late that the real value of their business is data. There has been a rush to create huge Data Lakes to store the enormous amounts of data available inside each company. The concept of a Data Lake is that of a low cost, but highly scalable infrastructure in which all types of data can be stored.
This sounds good, but creating a Data Lake is not easy and a good design is a must.
Although the basic definition of a Data Lake is a storage repository that holds a vast amount of raw data in its native format, it is also much more than that. The core function of a Data Lake is storage, but it is also being able to process this data rapidly in the same place so that different users may access it from the same location later.
Data can come in different formats: raw or structured. Depending on the user, it may be wise to store the former in distributed file storage, such as HDFS or S3. Raw data can be handled by a user with technical knowledge -a developer or data scientist- but business users will find it more difficult. In the latter case, (semi-)structured data in SQL or NoSQL databases would be better choice.
A computing framework should be placed around the data in order to read, transformand create new data. To avoid system redundancy it is recommendable to use an all-in-one solution such as Apache Spark, which permits batch, SQL, Streaming and Machine Learning operations within the same platform and API.
To be able to monetize the data, new applications and services will be created on top of the Data Lake. A good design is therefore important to ensure flexibility.
A Data Lake is valuable for the data it stores, which can come from a high variety of sources. These can be divided in two types: Internal and External sources.
Internal sources are the easiest and generally most-used ones. All data comes from inside the company so the cost will be lower than for external sources. Here are some examples:
• CRM: A Customer relationship management (CRM) is a system for managing a company’s interactions with current and future customers. It often involves using technology to organize, automate and synchronize sales, marketing, customer services and technical support.
• Website Tracking: Tracking all users using a single system instead of searching within the logs can have many benefits. It requires a small JS script on the company websites and a simple collector server.
