Improving the Quality of Data on Hadoop –
- by 7wData
As the value and volume of data explodes, so does the need for mature data management. Big data is now receiving the same treatment as relational data -- integration, transformation, process orchestration, and error recovery -- so the quality of big data is becoming critical.
Because of the promise and capacity of Hadoop, data quality was initially overlooked. However, not all Hadoop use cases are for analytics; some are driving critical business processes. Data quality is now a key consideration for process improvement and decision making based on data coming out of Hadoop.
With the size of our data stores in Hadoop, we must consider whether data quality practices can scale to the potential immensity of big data. Hadoop obviously shatters the limits of data storage, not only in terms of data volume and variety as well as in terms of structure. One way that data quality is maintained in a conventional data warehouse is by imposing strict limits on the volume, variety, and structure of data. This is in direct opposition to the advantages that Hadoop and NoSQL offer.
We must also consider the cost of poor data quality within a Hadoop cluster. From an analytics perspective, "bad data" may not be as troublesome as it once was, if we consider the statistical insignificance of incorrect, incomplete, or inaccurate records. The effect of a statistical outlier or anomaly is reduced by the massive amounts of data around it; the sheer volume effectively drowns it out.
In conventional data analysis and data warehousing practice, "bad data" was something to be detected, cleansed, reconciled, and purged.
Â
[Social9_Share class=”s9-widget-wrapper”]
Upcoming Events
Shift Difficult Problems Left with Graph Analysis on Streaming Data
29 April 2024
12 PM ET – 1 PM ET
Read MoreCategories
You Might Be Interested In
How Digital Transformation is Driving Customer Experience?
12 May, 2019It has been years since computers had been introduced into the world. Fact, we have become familiar with modern technology …
Predictive Analytics: Intersection of Humans, Machines
12 Jul, 2017Putting all your eggs in one cloud basket is risky, because clouds are not immune to denials of se Is …
BI and Data Science: Deliver Insights Through Embedded Analytics
9 Apr, 2021Data Scientists are trailblazers. They look for value inside of data and seek to ask the right questions, disseminating insights …
Recent Jobs
Do You Want to Share Your Story?
Bring your insights on Data, Visualization, Innovation or Business Agility to our community. Let them learn from your experience.