How Facebook Accelerates SQL at Extreme Scale
Serving SQL queries on a petabyte of data is one thing, but delivering it at Facebook’s scale is something else entirely. Earlier this year, the social media giant implemented the Alluxio distributed file system into its massive data architecture to speed up queries, while maintaining the separation of compute and storage.
Facebook has long been a proponent of open source software, and it has generated its share of open source projects in the big data space. Apache Hive and its follow-on, Presto, both emerged from Facebook, as did RocksDB and Apache Cassandra. GraphQL is one of a number of developer tools that came out of Facebook, along with PyTorch.
Facebook was an early adopter of Apache Hadoop, and remnants of Hadoop are still evident in the company’s architecture. It has moved away from storing data in Hadoop clusters, but it still uses an HDFS to access data in Warm Storage, which is the custom distributed file system that Facebook developed for itself after it moved off Hadoop. Warm Storage functions as a BLOB store atop a huge number of spinning disks, not dissimilar to Google Colossus or Amazon S3, Facebook Software Engineer James Sun tells us, and it uses synchronous Reed-Solomon error-correction coding to bolster data resiliency at massive scale.
The separation of compute and storage is a pillar of modern computer architecture, and is something that many organizations are striving toward today. But for Facebook, separating compute and storage is an absolute necessity, as the costs due to wasted storage or compute capacity would quickly add up.
Internally, Facebook is a big user of Presto for powering ad-hoc interactive SQL queries, A/B testing, and for serving critical dashboards, such as daily active users (DAU) and monthly average users (MAU). As a compute engine with no storage component, Presto is a great fit for companies that have separated compute from storage (that is why Facebook invented it, after all).
While Presto has worked well for Facebook, physics has not. To keep SQL query latencies in a workable range (i.e. around a second), several years ago Facebook implemented a data caching strategy. Dubbed Raptor, the system requires users to build ETL pipelines to cache data on SSDs attached to the Presto cluster.
Raptor delivered the query speedup that Facebook needed, since many Presto queries no longer needed to travel across the network. But Raptor created other problems along the way.
For starters, by hooking compute and storage back up together, it eliminated the company’s ability to scale both separately, which is bad. It also led to fragmentation of the data and a degraded user experience, because sometimes the queries were hitting the giant Warm Storage data lake and sometimes they were hitting data cached in the local SSD, according to Sun. It also complicates security and privacy, because now Facebook engineers have to worry not just about protecting data in Warm Storage, but also data sitting in these local SSDs.
Last fall, Facebook started to implement a replacement for Raptor. Based on a customized version of Alluxio, the new system delivers a comparable query performance as Raptor, but without the necessity to cache data on local SSDs.
Alluxio is a virtual distributed file system that serves as the memory-based glue to connect multiple computational engines with multiple backend storage systems. It was created by Haoyuan “HY” Li at UC Berkeley AMPLab in 2013.
