How Growth Street built its business intelligence stack
There are A LOT of ‘business intelligence’ tools out there. It can be hard to find guidance on what you need and how to tie different tools together. This post will explain how we think about BI at Growth Street. We cover the analytics stack we decided on and go through our decision-making process.
The graphic below from Matt Turk and Jim Hao at First Mark paints a picture of what is a crowded and sometimes confusing BI landscape:
There are some big questions to answer when first setting up analytics infrastructure. What type of database do I want? How am I going to get data into my database? Can I choose a BI tool that will sync my data for me? Do I need a self-service BI tool? How am I going to transform my data?
When this project started last year, we had a blank canvas and a long, long list of questions. After initial discussions, a few themes started to emerge:
We decided on a vision for business intelligence infrastructure that looked like this:
This type of stack closely matches with the one outlined by Tristan Handy at Fishtown Analytics. I’d worked with Fishtown before, and Tristan’s work influenced the infrastructure we ended up with.
We needed to understand all the different data sources that we needed to sync:
We needed a tool that would integrate with as many of these data sources as possible, which led us to…
The next step was to select an ETL tool. ETL stands for Extract, Transform, Load. It’s the process where data is moved from data sources to your data warehouse. This process often includes transformations — any kind of manipulation of the data — along the way.
ETL work is by far the least sexy part of an analytics stack. It often gets prioritised below product development activity. ETL tools have historically been expensive with clunky UIs, and a bit buggy and time-consuming. Luckily, some new players have emerged to take the work off your hands. The most notable players are Segment, Stitch and Fivetran.
It’s worth noting that each of these tools only does the extract and load portions of the ETL process. Businesses are increasingly choosing to transform and model after the data is in the warehouse. Data warehouses have become much cheaper and much more powerful. There is also an increasing number of great tools to transform the data within the warehouses.
The offerings of the three tools above have been converging over the past couple of years. To cite Stephen Levin’s very good comparison of the three, the short summary is:
We opted for Stitch. It covered all our data sources and integrates with all the well-known data warehouses. It serves almost all our needs and is the option that we chose for our third party data sources.
Stitch has also been great for the less common data sources that we have come across. Their team developed an open-source tool that allows end-users to build their own Stitch integrations. We have done so on two occasions and are currently building our third custom integration.
While Stitch could manage our PostgreSQL databases, we needed higher latency syncs for some of our data. Instead, we are using AWS Database Migration Service (DMS), which has worked a treat. Many organisations will be in a situation where there isn’t one ETL/ELT tool that suits all their needs. We will continue using more than one for as long as it’s cost-effective and not cumbersome to manage.
The data warehouse is the engine room of the analytics stack. It’s where the build of our data transformation occurs. We needed something well-supported, cost-effective, and scalable.
The three most common options are AWS’s Redshift, Google’s BigQuery, and Snowflake. There wasn’t a massive deliberation here. We chose Redshift because of existing familiarity, cost, and all our existing infrastructure being in AWS.
Redshift and DMS are the only two portions of Growth Street’s analytics stack that our Engineering team manage.
