Common data lake challenges and how to overcome them

Businesses have the perpetual problem of trying to get a grip on their performance. Executives need to have the latest information on their revenue, costs and profitability. They also want these figures segmented by business unit, geography, product line and customer.

The problem is that getting this overall picture is difficult. The typical large company might have several hundred applications deployed globally to capture sales, logistics and supplier data. Customer and product data are scattered across these applications, often with conflicting or inconsistent classifications.

Corralling all this data and making sense of it has been a thorny problem for decades. Traditionally, companies took copies of key data from their transaction systems, amalgamated them into a corporate data warehouse and resolved inconsistencies in definitions by matching up inconsistent sales or product hierarchies as data was loaded into the data warehouse. The data then went through some data cleaning and was funneled into a carefully designed schema and stored in a relational database. Subsets of the database could be spun out into local data marts to satisfy the needs of a specific business unit. The data modeling and cleaning took time and scarce technology skills, and the carefully designed database schema was inflexible. If the company acquired another firm, it could take months to adapt the data warehouse schema to deal with the data of the newly acquired company. This inherent time lag meant business users would not always have the up-to-date data they required. Many of them circumvented the IT department and created data feeds they could control.

One of the most prominent data lake challenges is sifting through the copious amounts of data. Web traffic, sensor data and the like could be an order of magnitude higher in volume than traditional sales data, and relational databases struggled to cope with the sheer amount of data, especially at an affordable price. More and more data came from outside the enterprise. Much of it was unstructured, such as documents and images rather than numbers. This pressure led to the development of big data file systems such as the Hadoop Distributed File System (HDFS), which were designed for very large-scale storage using inexpensive commodity disk storage.