Building a data lake takes meticulous planning –

Data lakes can give organizations more freedom on storing and analyzing data than they get from traditional data warehouses. But building a data lake architecture also presents IT teams with a raft of challenges.

TDWI data management analyst Philip Russom detailed the potential benefits and pitfalls of data lakes in a webinar this month; he also offered advice on priorities and best practices for a data lake implementation, highlighting things such as the need to tie it to real business issues and ensure that solid data governance processes are in place.

For example, Russom said data lakes — which typically involve Hadoop and other big data platforms — let data scientists and business users analyze data that wasn’t accessible to them before: call center notes, social media posts, internet clickstream records and more. But, he added, the broader exploration enabled by data lakes has to be done “with an eye to getting real business value from this data.”

To aid in that process, organizations need to be careful not to scrub away useful information in the raw data collected in a data lake, Russom said. That requires a different approach from the way that structured transaction data stored in a data warehouse is cleansed and consolidated before being made available for analysis.

“If you cleanse the data, if you get rid of anomalies, if you standardize the data so it all looks the same, you may lose some of the stuff that you’re looking for,” Russom said. As examples, he cited identifying customer segments for targeted marketing and detecting possible fraud in financial transactions, which might be overlooked if outliers in data sets are eliminated.

The webinar was based on a TDWI report released in March that included survey data on how companies are using data lakes and what benefits or drawbacks they’re seeing. In the survey, conducted late last year, 23% of 252 respondents said their organizations were already using a data lake, while another 24% expected to have one in production in the next 12 months.

Russom outlined a list of 12 priorities for businesses implementing a data lake architecture. His tips can be condensed into these three main points:

Plan your data lake carefully, according to the specific needs of your organization.