Automating Data Analysis Is a Must for Midsize Businesses
Midsize company leaders are right to be excited about the opportunities for harnessing the value in their large datasets. But the data in midsize companies tends to be messy — spreadsheets and plain-text files, many in different formats, are difficult (if not impossible) to integrate. It takes a lot of time and money to clean it up to make it useful. Poor-quality, disintegrated data can sabotage even the best initiatives, including AI designed to increase value and efficiency. HdL Companies, a Brea, California–headquartered government services firm, used their data strategically and has seen significant efficiency gains. The author offers three lessons for leaders to consider when getting started with automating data analysis.
As midsize companies grow, they develop data flows and data lakes (repositories for both structured and unstructured data) that are too big for one person, or even a team, to manipulate and use effectively. And even if a company is currently deriving value from its data, the people doing the work might move on, leaving the business tasked with having to find, attract, and hire expensive data analysts in a hurry.
Having a capable, up-to-date enterprise resource planning system (ERP) won’t solve the problem or relieve the pressure. Most midsize companies begin with finance-focused ERPs and wind up bolting on systems to store other data, such as customer activity and manufacturing throughput — a move that’s more operational than strategic.
Consequently, automating data analysis as the business grows is a very, very good idea. Automation is often where programmers write algorithms that perform previously manual tasks as instructed. Doing so pays dividends quickly, drives innovation and more growth, and paves the way to implementing artificial intelligence, which makes just about everything easier and more efficient and cost-effective. AI is coded to learn to perform a task, in some sense inventing and writing its own algorithms.
But the data in midsize companies tends to be messy. Spreadsheets and plain-text files, many in different formats, are difficult if not impossible to integrate. It takes a lot of time and money to clean them up to make them useful. Poor-quality, disintegrated data can sabotage even the best initiatives, including AI designed to increase value and efficiency.
As Joe Pucciarelli, group VP and IT executive advisor at the market research company International Data Corporation (IDC), said in a recent Channel Company webinar, “Most organizations’ data sets are not in great condition. We talk about data and analytics as a strategy and priority, but the data isn’t ready to support it.…Most organizations, when they’re trying to solve a problem, the analyst who’s working on it typically spends 75%+ of the time…simply preparing the data.”
As you might imagine, the ROI on the time spent doing that is not good. Let’s look at how one midsize company harnessed the value in its data and explore three steps midsize business leaders can take to do the same.
One of my clients, HdL Companies — a government services firm headquartered in Brea, California — is engaged by municipalities in California, Texas, and other states to analyze their respective states’ distribution of sales tax revenue to ensure that their city or town is getting its fair share. HdL looks for misallocations and discrepancies that municipalities can point to when petitioning the state for redress. The heart of this work is comparing different databases to expose discrepancies that affect who should get sales tax revenues. For example, in one database a business might be listed in Dublin, CA, but in two other databases it could be listed in neighboring Pleasanton. That makes a tax-allocation error highly likely; HdL’s job is to ferret it out.
California’s 40 million residents buy taxable products from 5.9 million licensed resellers, creating a massive data set of nearly 46 million tax records in 2020.
