There might be a few organizations that know exactly how many servers they’ll require to handle their database needs for any given period of time, but for most large enterprises whose success depends upon flows of multiple terabytes of data in often unpredictable surges, this is rarely the case.
Capacity planning is a guessing game, especially when an organization’s lifeblood is real-time, streaming data that enables it to perform real-time analysis to deliver interactions to users. The safest decision is to pay for peak capacity—even during times when far fewer servers are needed.
Why haven’t enterprises benefited from the cloud’s innate elasticity when it comes to database provisioning, and why is it getting even more painful?
In this post, I’ll discuss the challenge of controlling soaring data costs, the hurdles that stood in the way of true consumption-based usage models for cloud-based database services—and how the advent of serverless data can improve the economics for cost-conscious, innovative businesses.
Data architectures have become increasingly complex as enterprises have needed to build highly responsive applications that can analyze data in real-time and tailor experiences to users (a personalized recommendation based on an item placed in a shopping cart, a smartphone alert that a car tire is low, or a real-time notification of a suspicious transaction in credit card account). With all of these data sources, IDC estimated that more than 59 zettabytes of data would be created, captured, copied, and consumed in 2020.
Rising, non-linear costs have accompanied this volume and complexity—and this has only been exacerbated by the pandemic that has forced many companies to accelerate their digital initiatives.
“Before the COVID-19 crisis, many organizations were projecting the need for more data investment, and the crisis has likely only increased this need. With bottom lines already under pressure from the pandemic’s economic fallout, businesses might wonder where they can find the resources to meet that funding requirement.”
In the report cited above, McKinsey projects that spending on data-related initiatives will increase an average of 50% between 2019-2021, when compared to 2016-2018. Data costs can account for the largest part of the budget when it comes to building an application.
The serverless concept isn’t particularly new; it’s commonly associated with AWS Lambda, Amazon’s serverless cloud computing service. While serverless data is relatively new, the basic principle of serverless applies similarly to compute and data.
Serverless is built on next-generation public cloud services that auto-scale and charge only when used. When scale, Capacity planning, and cost management are automated, the result is software that's easier to build and maintain, and significantly cheaper.
But databases have proved challenging to make serverless. Modern databases store data by “partitioning” it across several nodes of a database cluster.
