Operationalizing AI — Managing the End-to-End Lifecycle of AI
We know that the Build-Run-Manage paradigm needs to integrate with existing CI/CD mechanisms to shepherd data science assets from Dev to QA to Production. It also needs to work with popular platform technologies like Docker and Kubernetes / OpenShift. We know that the Manage paradigm needs to establish correlations between model metrics and business KPIs.
Let us take a closer look at each of the five areas.
A data science team is a precious resource for any organization, but they’re often swamped with use case requests from across the company. How can the team scope and prioritize the requests to ensure optimum use of the team?
They start by exploring business ideas in detail, insisting on clarity around the business KPIs. Without that clarity, a team is far too likely to judge the success of a project by the performance of the model — rather than by its impact on the business. Consider an example. A company’s HR department wants to use AI to predict which managers will be high performers. In this case, the business metric might be something outside the feature set and model output — say, a business KPI like employee attrition rate. How will the data science team capture this metric if it’s neither a model input nor a model output? How will they hope to correlate model performance metrics like precision/recall to this external business KPI?
Too often, a business captures its KPIs in emails, slides, or meeting notes, where they’re impossible to track, especially as they change. Insist on capturing and understanding business KPIs up front (even if they change later). Doing so allows the data science team to prioritize and create swim lanes for each use case they undertake. Later in the lifecycle, these KPIs will need to be evaluated and correlated to model performance.
We all know the adages, “There is no Data Science without Data” and “Garbage In, Garbage Out”. It’s clear that good data stands at the heart of any successful AI project. But it’s still worth asking whether the data science team fully understands the data sets they’re dealing with. Do they understand metadata and how it maps to a business glossary? Do they know the lineage of the data, who owns the data, who has touched and transformed the data, and what data they’re actually allowed to work with?
On one occasion, we were working with a client data science team on a fraud detection use case. Our models performed well on hold-out data but performed poorly in production. We considered whether we needed to re-train the model to catch new fraud patterns, but no: something else was fundamentally wrong. Much to the surprise of all, we learned that we’d been working with data generated using rules, not ground truth, despite assurances from the data provider. It was a failure to understand and insist on data lineage — and a demonstration of how important it can be to work with a data steward.
Empower your data science team to “shop for data” in a central catalog — using either metadata or business terms to search, as if they were shopping for items online. Once they get a set of results, give them the ability to explore and understand the data — including its owner, its lineage, its relationship to other datasets, and so on. Based on that exploration, they can request a data feed. Once approved (if approval is needed), make the datasets available via a remote data connection in your data science development environment. As much as possible, respect data gravity and avoid moving data. If the team needs to work with PII or PHI data, establish the appropriate rules and policies to govern access. For example, you can anonymize/tokenize sensitive data. This stage is really about understanding the data you need for your AI initiative within a context of smooth data governance.
Most data scientists relish the build phase of the AI lifecycle — where they can explore the data to understand patterns, select and engineer features, and build and train their models.
