6 Keys to Big Data Victory
When I start helping a company unlock the vast potential of big data, I often get a tour of the technology they’ve acquired and data repositories bursting with potential. But frankly, that’s not what I want to see because shiny new technology and boatloads of data are not the keys to success.
After the tour, I ask about what really matters. Do you have the needed skills in your organization? (Usually the answer is yes, but not nearly enough.) Is there an established way for all the people with needed skills to work together? (Almost always, the answer is no.)
The power of big data emerges when a company assembles a cross-functional tiger team that works together in new ways to discover value in big data, operationalize their discoveries, and, based on a deeper understanding, implement innovations in the core business.
If you look at big data as something that happens inside your organization’s current boundaries, that uses technology to process data and discover signals, and then to deliver those signals in established ways, you will get some value. Bigger wins come from using big data and a data lake to support a much wider and deeper effort. This expansion can only happen when users (whether analysts or product engineers), software engineers, data management professionals, DevOps, and business experts all work together.
A data lake is perfectly suited to enable such a cross-functional team to thrive. Instead of seeing part of the picture, you can see all of it, going years back, in great detail, illuminated by powerful analytics. When the group has a question, the tools to massage the data, perform analytics, and create applications or deliver the data to existing tools are right at hand. Everyone can use their skills to quickly knock down barriers to progress. The data lake becomes the campfire around which everyone huddles and communicates.
Contrast this image of cross-functional creativity with silos, competing agendas, meetings, distractions, and politics. Of course, big data and Hadoop don’t eliminate these challenges with the wave of a wand, but the excitement and potential around them can spur creation of an effective cross-functional team, which is the secret ingredient in the most impressive victories I’ve seen.
I try to help the companies I work with understand the importance of cross-functional teams and a fully functional data lake. Here are 6 important tips to help you overcome common problems I’ve seen.
The goal can’t be something trivial or no one will care. It can’t be too large because you won’t get there. Create a roadmap that starts with achievable, impactful products that make substantial progress toward building a data lake and leveraging it. Get a small initial cross-functional tiger team together.
