Databricks wants to replace messy data lakes with more reliable ‘Delta Lake’

2 min read
Curated from siliconangle.com →

Big data firm Databricks Inc. wants to clean up companies’ messy data lakes with a new open-source project.
Delta Lake, as the project is called, acts similarly to a regular data lake but provides greater reliability by ensuring all of the information stored within it is “clean” and without errors, Databricks said.

Data lakes are systems or repositories of data stored in its natural format, usually object “blobs” or files. They usually act as a single store of all enterprise data, including raw copies of source system data and transformed data used for tasks such as reporting, visualization, analytics and machine learning.

But Databricks said the information stored in traditional data lakes can be unreliable or inaccurate for several reasons. These include failed writes, schema mismatches and data inconsistencies, which arise when batch and streaming data is mixed together.
“For the last decade, organizations have been building data lakes but have been failing to gain insights from the data, Databricks Chief Executive Officer Ali Ghodsi told SiliconANGLE. “Because it is garbage in – garbage out, organizations run into issues with data quality, scalability and performance.”

This unreliable data can prevent companies from deriving business insights in a timely fashion, and also slows down initiatives such as machine learning model training, which require accurate and consistent data, the company said.
“Delta Lake addresses these challenges by ‘filtering’ the messy data and blocking access into the Delta Lake,” Ghodsi added. “The clean data sits in a Delta Lake on top of the data Lake. This level of data reliability is not offered in today’s data lakes.”

Delta Lake ensures data is kept accurate and reliable because it manages transactions across batch and streaming data, as well as multiple simultaneous writes. Companies using Apache Spark to analyze their data can tap into Delta Lakes as their main information source, so they don’t need to make changes to their data architectures. In addition, Delta Lakes do away with the need to build complicated data pipelines necessary to move information across different computing systems. All of a company’s information can be stored in a Delta Lake, and hundreds of applications can tap into it as necessary.

Delta Lakes also make life easier for individual developer.

Continue Reading

Enjoyed this summary? Read the complete article at the source:

Continue at siliconangle.com →

Yves Mulkers

Yves Mulkers is the founder of 7wData and a widely followed voice in the data and AI community. He curates the 7wData and AI Beat newsletters, reaching hundreds of thousands of data and AI professionals, and writes on data strategy, analytics, AI, and the evolving data ecosystem.