To Centralize or Not to Centralize Your Data–That Is the Question
Should you strive to centralize your data, or leave it scattered about? It seems like it should be a simple question, but it’s actually a tough one to answer, particularly because it has so many ramifications for how data systems are architected, particularly with the rise of cloud data lakes.
In the old days, data was a relatively scarce commodity, and so it made sense to invest the time and money to centralize it. Companies paid millions of dollars to ensure their data warehouses were filled with the cleanest and freshest data possible, for historical reporting and analytics use cases.
As the big data boom unfolded, companies gained more options. Open systems like Hadoop let companies store petabytes of data on commodity hardware, spurring the creation of massive data lakes that held the less structured “data exhaust” at the heart of so many big data initiatives. Database innovation has also been steady, particularly around NoSQL databases that relax the rigid constraints of relational database to simplify life for developers, as well as NewSQL databases that replicate the time-tested reliability and durability of relational databases in the distributed realm.
The public cloud has been a force for both centralization and decentralization. On the decentralization side of the ledger are the multitude of software as a service (SaaS) applications, like Salesforce, NetSuite, and ServiceNow, which are extremely popular among Internet natives and brick and mortar companies alike. Since SaaS applications come with their own databases, it eliminates the need for customers to own and manage their own data store (which can be either good or bad).
On the other side of the coin are object storage systems like Amazon Web Service’s S3 and Microsoft Azure Data Lake Storage (ADLS), which provide very scalable and very inexpensive places to park vast amounts of data. Moving up the cloud stack, we see databases like RedShift, Google Cloud‘s BigTable, and Snowflake, which essentially recreate high-performance SQL data warehouse systems in the cloud, but with the added advantage of separating compute and storage, which lets users scale both independently and is a product of today’s containerized cloud architectures (thank you, Kubernetes).
Companies also have the option of building cloud-like systems on prem or in hybrid configurations, such as the Cloudera Data Platform (CDP) products, which provide many of the architectural advantages of cloud-based systems, but without the lock-in that comes with exclusive use of the big cloud providers’ data storage and analytic offerings.
In other words, companies have a plethora of options today when it comes to managing their data, and it’s not always clear how to straighten them out. So what should a data-driven company do? While every situation is different, there are some advantages and disadvantages to keep in mind for advanced analytics and AI use cases when choosing whether to invest in data centralization or leaving the data where it lays.
Sri Ambati, the CEO of H2O.ai, has watched data ebb and flow in the corporate data center over time. “Going back to the 1990s, companies with the best warehouses were able to mine it and get the most insights,” he says. “I think there’s always going to be value for centralization.”
Having all the data in one place can make it easier for a company to explore their data, to create features out of it, and to add more dimensions, Ambati says. But the track record in building modern centralized data systems is mixed. “The data lakes of the bygone big data era, if you will, have become data swamps, because people have not been able to use that effectively,” he says.
For many of today’s AI use cases, companies just won’t get a good return on their centralization investment, he says. “The reason for centralizing data is to run compliance processes, to run analytics, to understand what to do better, what to do next,” he says.
