In today’s world, it has become evident to almost everyone that data is growing at an exponential rate. However, it seems that only data experts are aware of what that exponential growth really means when it comes to storing and organizing all that data. The simple fact of the matter is that data is now everywhere, being constantly generated by most any electronic device, ranging from IoT sensors to personal smartphones.
Of course, that data needs to be stored somewhere, or it becomes lost to the world, never to be accessed again. There seems to be no end in sight for the growing flood of data being created. Even so, I am still amazed by the fact that it is estimated that the amount of data produced in the last 5 years by humanity is over 100 times bigger than the whole aggregated amount of data in the entire human history.
I’m not one to suggest discarding data in favor of reducing the amount of storage needed. It is pretty evident that data has value, and even the most esoteric pieces of data can be used to tell a story, yet that leaves us with a conundrum. How do we store and manage this ocean of data?
Research organization Statista offers some sobering estimates, which are clearly defined in the chart below, which gives an indication of the exponential growth of the amount of data created, captured, copied, and consumed, and its forecast up to 2025.
I find the results of Statista’s research both intriguing and somewhat scary, especially in light of that only just around 20% of that data generated is stored. Yes, just 20% is stored and 80% is disposed of or lost. It is forecasted that by 2025 the world will be able to store up to 100 zetabytes of data, which means only around 55% of the data generated in that same year will be preserved.
Humans, with the aid of technology, are creating massive amounts of data, with a significant portion of that data being lost, since it appears to be inconsequential. Yet, all data has some type of value, especially in the era of Artificial Intelligence. AI has a hunger for data and the more data available to AI, the better. AI analyzes that data to optimize applications.
However, storing all of the data ever generated remains a significant challenge.
