Why It Might Be Time To Start Looking at Small Data
Much of the buzz around data over the past few years has been about big data and how organizations are using large data sets to give them a business edge. However, due in no small part to the pandemic, small data is moving into the spotlight.
According to Bryan Philips Cupertino, Calif.-based In Motion head of marketing, small data is the opposite of big data. It is a term that describes data sets with fewer than 1,000 rows or columns. The term was coined in 2011 by researchers at IBM to describe datasets that are too small for traditional statistical methods. In contrast to big data, small datasets can be analyzed using estimation. Examples of small datasets include customer transactions, social media posts, and individual genome sequences.
Small data, or the use of small data sets is not new. In 2019, Arun Ramaswamy, Chief Technology Officer for NielsenIQ, pointed out in a CMSWire post that the era of big data is coming to end. He wrote that it will be replaced by small data sets as AI is developing so that it can do more with less and because it is harder to get consumer data access because of the emergence of regulations like the GDPR and the California Consumer Privacy Act.
But small data goes back even further then that. In fact, as early as 2014, David Lavenda, a product expert with extensive experience leading information-intensive technology organizations, wrote that while specialized business analysts have been able to exploit it at a macro level, big data has failed to provide individual workers with the insights they need to act daily.
“There is nothing wrong with big data per se; but it’s not actionable for individual workers. What workers need is not big data, but rather, relevant data presented in smaller and smarter chunks,” he wrote in CMSWire.
It has clearly taken a long time for this message to trickle down that small data is useful too, but it has and now Gartner has identified it as one of the top 10 trends in the data and analytics space for this year. It is hard to know if the pandemic is the cause of the change in focus, but it certainly contributed to the change.
“The speed at which the COVID-19 pandemic disrupted organizations has forced D&A leaders to have tools and processes in place to identify key technology trends and prioritize those with the biggest potential impact on their competitive advantage,” said Rita Sallam, distinguished research vice president at Gartner, in a statement about the trends.
The research also points out that with the evolution and widespread traction of artificial intelligence (AI) and machine learning (ML) across the enterprise, businesses can now apply new techniques for smarter, less data-hungry AI solutions.
More to the point, Gartner points out that the extreme business changes from the COVID-19 pandemic caused ML and AI models based on large amounts of historical data to become less relevant. Running parallel to that is the fact that decision making by humans and AI are wider and require data from different sources for accurate responses to queries.
As a result, Gartner recommends that organizations adopt technologies that can use whatever data is available, as well as, wider sets of data that enables the analysis and use of synergy of a variety of small and large, unstructured, and structured data sources, as well as small data which is the application of analytical techniques that require less data but still offer useful insights.
Small data analysis also enables small enterprises to join the data party too, according to Lior Shamir, associate professor of computer science at Kansas State University. He said that small data is data that humans can read and understand without the need to use machines.
