If you’d asked an IT professional whether object storage is any good for databases, the answer over most of the past decade or so would have been a resounding “no”.
The response would have been pretty obvious because databases, especially in busy and mission-critical environments, are subject to a lot of changes from a lot of users, either simultaneously or almost simultaneously.
Databases need IOPS (input/output operations per second) and they need some way of enforcing consistency of data, meaning block access SAN storage has long been the way to deliver that.
However, that “no” answer may have changed over the past few years to something more like a “maybe”.
With the rise of databases that use in-memory storage, IOPS are in plentiful supply close to compute, so it has become possible for object storage to be the site for bulk storage of datasets, with segments moved to memory during processing.
But does this constitute database operations using object storage?
For a couple of decades, SAN block access storage was the go-to for running databases and the enterprise applications built on them. IOPS was king, as potentially numerous I/O requests hit the database from client systems. To cope, SANs got ever larger and more performant, with the eventual mainstream adoption of very quick – in IOPS terms – flash media. NetApp’s current advice on storage sizing for SAP HANA, for example, is in terms of (presumably on-site) SAS HDDs and flash capacity. But while SANs can deliver in terms of IOPS – even if that becomes partially redundant as application databases go in-memory – they still have their limits, and that is in terms of scale. Here, object storage excels. It cannot produce the kind of IOPS that a SAN can provide to database operations, but it can give throughput in large volumes. There are two good reasons as to why that’s a big deal right now. One, for several years the volumes of data being analysed have grown ever larger. With block or file access storage starting to become unwieldy above 100TB, object storage looks like a good bet with its ability to scale to PBs. At the same time, object storage has become the de facto storage mode for the cloud, adding to its prevalence both off-site and on-site. In addition, as part of the backdrop, we’ve seen the emergence of in-memory database-based applications such as SAP HANA delivered in the cloud. A big benefit of cloud object storage is the low cost. On Amazon, for example, file or block storage can cost 10x more than object storage.
With the emergence of object storage, and in particular S3, we have seen the rise of its use as bulk storage that can be delivered to in-memory database work and for AI/ML analytics. Alongside this trajectory has been the emergence of databases that will work with S3 (or S3-compatible storage) as a data store, such as MongoDB, CockroachDB, MariaDB and Teradata. Cloud data warehousing phenomenon Snowflake is also S3-based.
