You Can’t Out-Architect Bad Data
Say it with me: bad data is inevitable.
It doesn’t care about how proactive you are at writing dbt tests, how perfectly your data is modeled, or how robust your architecture is. The possibility of a major data incident (Null value? Errant schema change? Failed model?) that reverberates across the company is always lurking around the corner.
That’s not to say things like data testing, validation, data contracts, domain-driven data ownership, and data diffing don’t play a role in reducing data incidents. They do. In fact, we’ve written extensively about emerging data quality best practices like data SLAs, circuit breakers, immutable/semantic warehouses, schema change management, data asset certification, and moving toward a data mesh.
John Steinmetz, VP of Data & Analytics at shiftkey, echoes this sentiment. We had the opportunity to interview him for our blog a few weeks ago on these and other tips for building out the first data team at your startup. Check it out.
As any data practitioner will tell you, throwing technologies at the problem is not a silver bullet for data quality, but they can be used as guardrails against some of the less obvious issues that wouldn’t otherwise be caught by diffing or testing.
In the wise words of our Site Reliability Engineering forebearers: “Embrace risk.”
In a greatpieceby dbt Labs CEO, Tristan Handy, he provides several helpful suggestions for tackling the upstream data issues that occur when software engineers push updates that impact data outputs in tightly coupled systems.
It’s a valuable response to one of our earlier posts on data contracts. While I agree with his overall thesis of trying to prevent as many data incidents from hitting prod as possible, I would quibble with one of his subpoints that:
“Probably, one of those expectations should be: don’t make changes that break downstream stuff.
Does that feel as painfully obvious to you as it does to me? Rather than building systems that detect and alert on breakages, build systems that don’t break.”
I don’t mean to single out Tristan (who regularly writes to share his expertise with the community) because this is a thought lurking in the subconscious of quite a few data practitioners. The idea that, “Maybe I can build something so well that bad data never enters my pipelines and my system never breaks.”
It’s an optimistic – and tempting – proposition. After all, data engineers, and data analytics professionals more broadly, need to have a certain “get it done against all odds” mindset to even decide to enter the profession in the first place.
The odds are against us in the number and diversity of ad-hoc requests we have to field on a daily basis; the odds are against us in the growing size – and scale – of our tables we have to manage; and the odds are against us when it comes to understanding the growing complexity of our company’s data needs.
Socrates, who said, “I know that I know nothing,” would have been a good data engineer. Image courtesy of Monte Carlo.
But the best data engineers and leaders understand two things: 1) they can’t anticipate all the ways data incidents can arise and 2) the consequences for these incidents are becoming more severe.
