Like any good story arc, we’ve come a long way since the origins of data analytics. The first phase of BI started with rigid, IT-owned systems. The second phase followed a wave of more flexible, business-oriented tools that enabled a more business-facing data Analyst mindset — and a tsunami of pretty, easy to filter, but still static dashboards.
Today — with the rise of cloud-native data warehouses and advancements in scalable inference methods — we’re at the cusp of a third phase that not only affords better, faster processing of data, but also lets operational data analysts impact business decisions like never before. I call this phase Data Analyst 3.0.
Before we look at the factors bringing in Data Analyst 3.0, let’s take a look at how far we’ve come. It used to be that a single person within the IT team could gain all the relevant domain and technology skills necessary to become a “data expert.” Data wasn’t big or wide, which meant that people could obtain new data skills (Excel, lightweight SQL, SAS, etc.) as problems arose, and the process of sending over a CSV to answer questions worked just fine.
But, from the organization’s perspective, most data requests failed in the handoff between IT and the business because technologists didn’t know how to make their data infrastructure consumable to an everyday Excel user. The queries that IT teams could deliver only answered a single question about a specific KPI. This had two major issues:
Fortunately, this system has largely disappeared over the last ten years alongside the rise of more business-centric data modeling, BI, and visualization tools. Think Qlik, Tableau, and Looker: the second wave of BI, and the preferred tools of Data Analyst 2.0.
Beneath these end-user tools, this second wave is supported by several platforms that make it easier to derive value from the vast amounts of data we’re storing. Collectively, these tools make up a typical analytics stack.
The exact evolution of this analytics stack is a fascinating topic, but I’ll save it for another post.
To navigate and maintain this stack efficiently, businesses needed more than just the IT team, so a few common roles emerged:
One way that you can think about the distinction in these roles is whether they act before or after the data is collected. Data Engineers are responsible for operations before the data is collected (and transformed), while Analysts and Data Scientists are responsible for operations after the data is collected.
Like Google’s Cassie Korzykov mentions in one of her insightful posts, if your primary skill falls closest to that of a Data Analyst, chances are you feel left behind in your “technical” expertise by your Data Science counterparts. Even the job market views the Data Scientist role as a level up from you. Only a few people realize that these two roles are entirely different from one another.
Data Scientists provide high-effort solutions to specific problems. If the issues they tackle aren’t worth solving, businesses end up wasting their time. They are narrow-and-deep workers, so it’s imperative to point them at problems that deserve the effort. To ensure you make good use of their time, you need to be sure you already have the right problem or need a wide-and-shallow approach to finding one.
This is where a Data Analyst can help the business. A Data Analyst’s primary goal is to surf vast datasets quickly, liaise with the business stakeholders, and surface potential insights. Speed is their highest virtue. The result: the company gets a finger on its pulse and eyes on previously-unknown unknowns. This generates the inspiration for decision-makers to select the most valuable quests for Data Scientists.
Unfortunately, most Data Analysts today are stuck in a quandary. They’re sitting on a treasure trove of rich, wide data, but they’re stuck combing through it with tools ill-equipped to handle such a big job.
