Big Data, Data Mining and Machine Learning: Deriving Value for Business

“Hiding within those mounds of data is knowledge that could change… the world.” – Atul Butte

We are currently living in a post-modern world, an age where technology, data, and information rule the world. Consequently, it is easy to believe that the concept of Big Data is a unique phenomenon that started in 2012 when Barack Obama (and his government) announced the Big Data Research and Development Initiative.

However, large amounts of data have been around since the development of the Internet in 1991. The fundamental difference between then and now is that our ability to interpret these masses of data has evolved to the point that we can now utilise this data as part of our business decision-making process.

Big Data and Data Mining: Collecting the right data

When the focus first shifted from data storage to the value of Big Data, it was easy to collect and store as much data to do with every aspect of running the business as possible in the likelihood that it might be used sometime in the future.

However, this focus has now shifted from simple data collection to the collection of relevant data; data that adds value to the business. Only collecting lots of data is not enough. Collecting data on a large scale gives you big data; thus, plenty of data; but it doesn’t necessarily mean that you have valuable data.

Useful data not only needs to be big data, but it also needs to be high-quality, practical information. In other words, companies need to collect data about each subject that is detailed enough to allow analytic tools and models can drill down into as much detail as required.

This is where data mining comes into the picture. Essentially, data mining is the methodology which is employed to sort through large data sets to identify patterns and relationships. These patterns and relationships are then used to solve problems and predict future trends.

The data mining methodology is only implemented once the raw data has been extracted, transformed and loaded into a data warehouse.

One of the simplest machine learning models is the recommendation engine.