Moving Thumbtack’s data infrastructure to Google Cloud Platform
In early 2015, we began building data infrastructure at Thumbtack. At the time, the company’s data was spread across a PostgreSQL replica and a MongoDB cluster, neither of which were scaling adequately to handle the load of analytics and BI queries. Our largest table in PostgreSQL had more than a billion records and many queries took nearly an hour to complete.
Furthermore, behavioral event data stored in MongoDB was inaccessible to most of our internal data consumers, particularly those without a technical background. Thus, a critical part of Thumbtack’s data had become “dark data.”
Separately, Thumbtack had an increasingly pressing need for more scalable data processing infrastructure. This was driven by a need for better A/B testing and our nascent machine-learning efforts.
To address the need for easier data access, we formed a team focused on (1) making all of Thumbtack’s data easily and reliably accessible, and (2) providing scalable infrastructure for large-scale data processing. Our first efforts involved a monolithic Cloudera CDH cluster on AWS EC2 machines. (All of Thumbtack’s infrastructure was on AWS at the time.) We then adopted Apache Impala to provide distributed SQL support, and Apache Spark for distributed data processing.
This configuration worked well for us for about a year before we began hitting some operational issues. In particular, running production Spark workloads alongside ad-hoc Impala SQL queries meant that we had to significantly over-provision the cluster hardware to ensure that we could reliably handle peak workloads. Our team uses Mode Analytics for dashboarding, and we found that most of our users would schedule reports to run daily at 8 or 9am before coming into the office — often overwhelming the cluster, which would still be finishing up nightly jobs started in the early morning hours. Resource contention also impacted our Spark jobs and we began staggering daily job starts throughout the night (when ad-hoc queries were the quietest) to ensure jobs had sufficient resources. Finally, as our cluster grew, operational issues like machine failures became a significant time draw for our team.
Towards the end of 2016, we started evaluating potential new infrastructure designs beyond a single monolithic cluster. At the time, we had dozens of Spark jobs and thousands of Impala SQL queries running daily against billions of records. Given our small team size, we prioritized two goals: (1) moving toward more managed services and away from VMs that we needed to manage ourselves; and (2) fully elastic resources to eliminate our need for a large excess of standing capacity.
For our next solution, we evaluated three options: continue expanding our monolithic cluster, move to AWS managed services or move to Google Cloud Platform (GCP). As we learned more, we found ourselves increasingly drawn to GCP — in particular, the combination of Google Cloud Storage + Google Cloud Dataproc (Spark) + Google BigQuery (SQL) was ideal for our use case. We began our migration in earnest in early 2017, and shut down the previous production cluster in May.
During the course of standing up our infrastructure on GCP, we learned a few tricks that really helped in making our new infrastructure successful. It was surprisingly easy to wire up GCP resources to our existing infrastructure — particularly with Apache Airflow, which we use for all of our workflow scheduling. A particular focus for us was getting to an MVP solution as quickly as possible, to (1) ensure that we were demonstrating clear value and (2) find and address roadblocks early on in the process.
One of the things we found to be really helpful was to partner with our analytics team to identify a list of the most critical datasets to make available. In our initial efforts, we focused on migrating those datasets to BigQuery.
