Why observability in dataops?
It’s 8 a.m., and a business leader is looking at a financial performance dashboard, questioning if the results are accurate. A few hours later, a customer logs in to your company’s portal and wonders why their orders aren’t showing the latest pricing information. In the afternoon, the head of digital marketing is frustrated because data feeds from their SaaS tools never made it into their customer data platform. The data scientists are also upset because they can’t retrain their machine learning models without the latest data sets loaded.
These are dataops issues, and they’re important. Businesses should rightly expect that accurate and timely data will be delivered to data visualizations, analytics platforms, customer portals, data catalogs, ML models, and wherever data gets consumed.
Data management and dataops teams spend significant effort building and supporting data lakes and data warehouses. Ideally, they are fed by real-time data streams, data integration platforms, or API integrations, but many organizations still have data processing scripts and manual workflows that should be on the data debt list. Unfortunately, the robustness of the data pipelines is sometimes an afterthought, and dataops teams are often reactive in addressing source, pipeline, and quality issues in their data integrations.
The book Digital Trailblazer goes about the days when there were fewer data integration tools, and manually fixing data quality issues was the norm. “Every data processing app has a log, and every process, regardless of how many scripts are daisy‐chained, also has a log. I became a wizard with Unix tools like sed, awk, grep, and find to parse through these logs when seeking a root cause of a failed process.”
Today, there are far more robust tools than Unix commands to implement observability into data pipelines. Dataops teams are responsible for going beyond connecting and transforming data sources; they must also ensure that data integrations perform reliably and resolve data quality issues efficiently.
Observability is a practice employed by devops teams to enable tracing through customer journeys, applications, microservices, and database functions. Practices include centralizing application log files, monitoring application performance, and using AIops platforms to correlate alerts into manageable incidents. The goal is to create visibility, resolve incidents faster, perform root cause analysis, identify performance trends, enable security forensics, and resolve production defects.
Dataops observability targets similar objectives, only these tools analyze data pipelines, ensure reliable data deliveries, and aid in resolving data quality issues.
Lior Gavish, cofounder and CTO at Monte Carlo, says, “Data observability refers to an organization’s ability to understand the health of their data at each stage in the dataops life cycle, from ingestion in the warehouse or lake down to the business intelligence layer, where most data quality issues surface to stakeholders.”
Sean Knapp, CEO and founder of Ascend.io, elaborates on the dataops problem statement: ”Observability must help identify critical factors like the real-time operational state of pipelines and trends in the data shape, “ he says. “Delays and errors should be identified early to ensure seamless data delivery within agreed-upon service levels. Businesses should have a grasp on pipeline code breaks and data quality issues so they can be quickly addressed and not propagated to downstream consumers.”
Knapp highlights businesspeople as key customers of dataops pipelines. Many companies are striving to become data-driven organizations, so when data pipelines are unreliable or untrustworthy, leaders, employees, and customers are impacted. Tools for dataops observability can be critical for these organizations, especially when citizen data scientists use data visualization and data prep tools as part of their daily jobs.
