Big Data as a Service, an Interview with Google’s William Vambenepe
Many of the big data technologies in common use originated from Google and have become popular open source platforms, but now Google is bringing an increasing range of big data services to market as part of its Google Cloud Platform. InfoQ caught up with Google’s William Vambenepe, who’s lead product manager for big data services to ask him about the shift towards service based consumption.
InfoQ: Hadoop, HDFS and HBase were inspired by Google’s MapReduce, GFS and Bigtable. How much does the service now called Bigtable differ from the internal platform of a decade ago?
William: Bigtable has gone through several major iterations in its lifetime at Google, driven from the evolving requirements from supporting Google’s major applications. In many ways, the Bigtable that is part of the bedrock at Google today is significantly different from the technology that was originally developed in 2004. For example, after its internal implementation, a significant amount of work was done on improving the 99th percentile latency which became a stronger and stronger requirement when Google started serving traffic out of the database. This drove a lot of work into diagnosing and grinding away the tail latency. Additionally, multi-tenancy within Google has been a significant challenge, and in offering the technology as an external service, a lot of work had to take place around isolation of all the different layers of resources which are utilized. One last note is that this service is offered through the completely open source HBase API/client, which is somewhat ironic, given that Bigtable was the original service, and there is a tremendously powerful client infrastructure internally. However, we think it was the right thing to do, since the HBase community is diverse and powerful and we want to continue to work together with this amazing ecosystem.
InfoQ: Over the last few years Google has talked about a bunch of internal data services such as Dremel, MegaStore and Spanner; are they now finding their way into services that anybody can use on demand?
William: Definitely, many of them are. Many of the services we use inside Google are extraordinarily useful outside of Google. A clear example of this is BigQuery, which exposes Dremel as a service, allowing users to analyze potentially enormous (petabyte-sized) datasets with SQL queries which typically execute in just a few seconds and require no cluster management from the user. Another example would be Cloud Datastore, which currently relies on Megastore to provide a NoSQL transactional database that can handle Google-scale data sets. Beyond those, Google Cloud exposes other internal tools which have been described in published papers. For example, Cloud Dataflow unifies two internal tools, FlumeJava (for batch processing) and Millwheel (for stream processing) to provide a unified programming model and managed execution environment for both batch and stream. And yes, we’re always looking at situations where other Google technology could be exposed as a service — and Spanner is definitely something that’s generated a lot of interest in this area.
InfoQ: In tech we very often talk about better, faster, cheaper – pick two. Is it possible that cloud based big data services will offer all three (versus do it yourself approaches using open source or products)?
Just a few examples: BigQuery performance at scale is unparalleled (we have customer queries which process several petabytes in a single query). Dataflow unifies batch and stream in one programming model and offers the most advanced semantics in the industry for stream processing (e.g. windowing by actual event time, not arrival time). Bigtable vastly outperforms comparable databases on read and write latency. Etc. And all those capabilities come as fully managed services, not at the cost of weeks of deployment/configuration/tuning.
