The benefits and challenges of augmented data discovery tools
Augmented data discovery is an emerging BI capability for automatically preparing and organizing enterprise data for self-service BI. This is particularly challenging for unstructured data from sources like email, social media channels, IoT feeds and customer service interactions.
Traditional BI tools have supported basic capabilities for joining, manipulating and transforming structured data. Augmented data discovery can build on these basic capabilities with augmented data preparation and automated pattern discovery for self-service BI, according to research firm Gartner Inc. Augmented data preparation streamlines processes for data profiling, managing quality, cleaning data, modeling, enriching and labeling metadata in a manner that supports reuse and governance. Automated pattern detection builds on traditional BI tools to support complex, large data sets with more than 10 columns.
Augmented data discovery focuses on providing insight for citizen data scientists. In Gartner’s view, these are similar but somewhat different to augmented data science platforms used for building data inference models that can be embedded into apps. Consequently, augmented analytics tools also tend to include natural language query and natural language generation features. This ease of access promises many benefits, but enterprises also face several challenges in making the tools work well in practice.
Augmented data discovery reduces the time and complexity of deriving valuable insights from new data sets, especially unstructured ones. Cognitive services are often used by these tools to scale more efficiently than manual processes. They can process up-to-the-millisecond data on the fly to instantly derive data, said Stephen Blum, founder and CTO of PubNub, a data management API provider. Gain insight on live conditions: “The ability to see and act on real-time conditions has only been available via very expensive, noninteractive dashboards that provide little value,” said Mark Palmer, general manager of analytics at Tibco Software.
Now, any user can utilize BI tools to visualize, understand and act on live IoT data, live geographic data or a live view of business transactions in just minutes. This makes real-time commerce and customer engagement possible. By applying AI and augmented data discovery, we begin to make algorithmic sense of unorganized data swamps. Act on insights faster: Traditional BI tools were good at using data to course correct the business with minor adjustments. Augmented data discovery promises to make it easier to discover new insights that could guide more radical and impactful changes. Micha Breakstone, co-founder and head of R&D at Chorus.ai, a conversational analytics service, has been experimenting with methods such as anomaly detection and covert pattern recognition to discover deeper insights and, in some cases, proactively.
“Additionally, actionability can be modeled across various predefined business dimensions to ensure business value of the insights,” he said. Turn data swamps into data lakes: Companies have used cloud storage and Hadoop technologies to store data sets in case they may be useful one day.
But without a clear goal, data management architecture or governance strategy, it’s easy for these data sets to grow out of control. “It’s become a data swamp, not a data lake,” Palmer said. “By applying AI and augmented data discovery, we begin to make algorithmic sense of unorganized data swamps.” Reduce technical hurdles: Augmented analytics reduces the burdens around data profiling and data preparation for preparing reports.
“Using augmented data discovery, more business users are able to discover data and gain insights from the data, even if they do not know how one data element is related to another data element,” said Gal Ziton, CTO and co-founder of Octopai, a metadata management platform. For example, augmented data discovery could automatically join multiple tables required to generate a report.
