Integrated vs. specialized: Which shines the brighter light on “dark data?”
Ingest, data prep, analysis, visualization and export. They’re all part of the Big Data analytics life-cycle. The good news? The market sports multiple product categories that handle each of these. The bad news: there are far fewer products that handle many of these areas together. It’s getting more complex too, as other big data lifecycle categories, like data lake management/data cataloging and Big Data operations/DevOps are emerging.
This may be frustrating, but it’s also understandable: if there’s an area of functionality that the market is neglecting, then funding a company to develop and provide that functionality makes sense. It helps the market, it helps the company and, hopefully, it helps the company‘s investors.
Does it help the customer, though? That’s harder to say. If you’re doing your data prep in one tool and your analysis in another, that’s a bit regimented, and the context switch may be jarring. Moreover, if your analysis reveals to you more prep work to be done, you are sent back to the data prep tool for a “do-over,” rather than embarking on iterative effort.
Case in pointRecently, I spoke to Bob Laurent, VP of Product Marketing at Alteryx on a related subject. The discussion began as an investigation into Mr. Laurent’s notion of “dark data:” data that you have in your possession but aren’t using, or analyzing, for the betterment of the business. As Alteryx has a focus on data preparation, the discussion began with talk of how to avoid dark data, by grabbing, processing and coalescing data sets, be they buried in log files, point-of-sale feeds or elsewhere, to get at intelligence that would otherwise be latent and obscured.
That’s all fair game. The nitty-gritty of digging for valuable data — then scraping the debris away and making it sparkle and shine for observation and analysis — is the bread and butter of every company in the analytics field.
It wasn’t really enough though, not for an interesting conversation and not for a Big on Data post.
Lifting the darkness But then, as if sifting for data sets, we stumbled on something really valuable. Laurent described a case study to me — one where a financial services customer used Alteryx to look at its trading terminal data, to find correlations between certain trading patterns and various abuse incidents (like money laundering, for example).
