High versus low-level data science
Here I describe a case study: a solution based on high-level data science. By high level, I mean data science not done by statisticians, but by decision makers accessing, digging into, and understanding summary (dashboard) data to quickly make a business decision with immediate financial impact. There is also a section on smart imputation techniques, with patentable, open intellectual property that we created after investigating this problem.
This article is articulated in three sections
I have discussed various breakdowns or categorizations for data science:
Here I introduce a new type of distinction: high-level versus low-level.
Most people think that data science is low-level data science only, but that’s not the case. Note that low-level data science is to low-level programming what high-level data science is to high-level programming. The low-level layer is more technical and more complex: it’s the layer on which the high-level rests. But the high-level layer requires different skills, including business acumen, leadership and domain expertise.
About the problem and data set
The problem studied here was solved by the decision maker (a data scientist), using high-level data only, that is, highly summarized data. The data scientist in question monitors carefully selected compound web traffic metrics (KPIs, usually ratios) and noticed a spectacular drop for one of these metrics
Access to highly granular (low-level) data was not easy to get, and dashboard summaries, carefully selected and crafted, were sufficient to detect and address the issue with a one-week turnaround, doing a number of tests described in the next section.
More specifically, we used the Google Analytic dashboard. We did not access granular metrics such as IP address, detailed log-file transactions, or summary statistics broken down by user agent / referral combinations (not available from the dashboard). But we did use session duration, number of pages, and conversions, per day per referral, probing the summary data sometimes 2-3 times per day to check the results of a number of tests and fine-tuning, in short to check and quantify impact on performance. Performance here is measured as the number of real (not bogus) conversions per click, or conversion rate. We also looked at conversion rate, per paid keyword per day, available from the dashboard. The statistics per user agent per referral would have been very useful, but was not available. The user agent alone proved very useful in a related problem.
The detection of the problem is straightforward if you monitor the right KPIs, as all of us data scientists should do: see Figure 1. It might even be made easier (earlier detection) if there is a system of automated email alerts in place, designed by high-level data scientists. It is always a good idea to have an email alert system in place, for core metrics impacting revenue. Note that in Figure 1, none of the variations between Day 1 and Day 2 are significant, except the one for AdWords – both sharp and statistically significant (because based on a large number of sessions).
It was initially believed that the performance drop (click conversion falling from above 20% to below 10%) was due to click fraud, though we did not exclude a reporting glitch initially. A reporting glitch was ruled out – as a contributing factor – when actual conversions, measured internally (rather than via Google Analytics), were also down. We also tested our own ads, simulating conversions, to see if they were properly tracked by Google. They were.
Anyway, we eliminated most major sources of click fraud: traffic from affiliates, and mobile traffic. We ended up with clicks from US-based IP addresses only, from Google.com only, and not from a mobile device. The pattern of poor performance continued nevertheless.
