Big Data in the Cloud: Reaching a Tipping Point

It may be time to rethink where you want to support a big data initiative.

Even though you could find plenty of cloud providers that offered data storage and analytics services to enterprise customers, early big data projects frequently took place in-house. This fact led to many failed big data projects that, in many ways, tarnished the image of what big data analytics can do.

But as more and more big data success stories continue to trickle out, a picture starts forming. That picture showed that big data analytics will indeed be the key to business success in a digitally transformed world — and that you’re more likely to succeed if your data and analytics happen in the cloud.

There is plenty of data to back up the fact that data analytics will play a huge role in the competitiveness of businesses in the coming years. What’s less understood is why big data is more successful in the cloud when compared to on-premises. In this article, we’ll explore those reasons, as well as point out why big data in the cloud has finally reached a tipping point in terms of enterprise adoption.

Most enterprise IT shops that attempted a big data project in house knew that it wasn’t going to be easy. But even with a great deal of planning and grit, projects failed because the project simply consumed too many hours and required skill sets that, at the time, were scarce. Even in 2017, data admins that are proficient in big data platforms such as Hortonworks or MapR can basically write their own ticket. Many projects failed to complete because they experienced significant “brain drain” as their big data platform talent moved on to greener pastures.

Understanding the role of the data scientist

While there were significant shortages on the infrastructure side, the real downfall of in-house big data projects was found on the analytics side. After all, this was really enterprise IT’s first foray into true, in-depth data sciences at such a scale.