How to succeed with big data on a tight budget

Big data means big dollars for many companies, but it doesn’t have to be that way.

Much of the work I do with large companies involves using big data to drive a competitive advantage; as such, data science is woven into the corporate strategy and deservedly requires a significant investment in time, money, and resources. But do all data science efforts need to drive a competitive advantage? No.

I’m not suggesting you repeat the mistakes of early big data adopters who threw money at big data with no real plan or goal. But I do believe data science can play a useful role without breaking the bank.

If you aren’t spending a lot of money on data science, then make sure to set your expectations low. For instance, if you’re looking at a strategic effort with a $2 billion upside where big data plays a large role, your investment could easily be $100 million or more per year.

On the other hand, if you want to experiment with big data, you could hire a data scientist and buy a few machines and tools for approximately $300,000 per year. (I’m basing these figures on costs in Silicon Valley.) To take an approach like this, you must think of data science as research and development; with an R&D effort, there are no strong or immediate expectations for a return on investment. But it’s not a frivolous investment.

For a pharmaceutical company, R&D is the foundation for how its business works—but there’s a lot of faith involved. This is the way you should approach your data science experimentation. It’s okay to start with a low budget if you’re willing to accept little or no initial benefit.

A smart way to financially ease into data science is to leverage your available functions. If you have a business intelligence and data warehousing team that manages your decision support systems, you should consider carving out a small segment for data science. I bet it won’t take long to find one or two talented data professionals who would like to experiment with data science.

Another area to consider is your Lean Six Sigma or continuous improvement function.