What A Research Firm Learned From Hundreds Of AI Project Failures

AI projects aren’t dying because of big problems, but rather it’s the small things that are causing projects to fail. It’s very easy to get excited at the beginning of an AI project when the future has all sorts of possibilities. But then the day-to-day realities of making AI projects work interfere with those fantasy visions and AI projects are left to languish, eventually failing. One of the biggest cases of those “death by a thousand cuts” for projects is that organizations fail to factor in the long term costs of the project. Leadership budgets for only the initial costs of the project, forgetting about maintenance and upkeep.
Research firm Cognilytica concluded in an analysis of hundreds of AI project failures that organizations are not understanding that AI project lifecycles are continuous. Organizations often budget for the first few iterations of a project, with all the need for data preparation, cleansing, model training, data labeling, model evaluation, and iteration, but fail to keep up that budgeting for ongoing iterations. Organizations also have to continually monitor models and data for decay, retraining the model as needed, and taking future expansions and iterations into consideration. This results in organizations having a misaligned or unjustified ROI for the AI project over time.
Considering the cost of continuous model iterationut how exactly do you go about doing this in a thoughtful way? The challenge for most organizations is that they treat AI projects as a one-time, proof-of-concept or pilot application, failing to consider setting aside funds, resources, and people for continuous model evaluation and retraining. As a data-driven project, AI is not a one time investment. People often don’t realize they’ll need to allocate money, people and resources to the continued iteration and development of a model once it’s in production.
If, in the beginning of the project, organizations only take into consideration the cost of building the first version of the model, they often run into issues. In considering AI project cost and ROI, AI project owners need to ask how much it’s going to cost you to maintain that model and how much they’re willing to invest in ongoing data preparation and model iteration.
One best practice learned from successful AI project owners is that AI projects aren’t one-time delivery of functionality. Rather, AI projects should be looked at as continuously iterative cycles, not a start point and finish line. In much the same way that cybersecurity projects aren’t one-time things, neither are data-driven projects, of which AI projects are a subtype. The real world continually changes and so does the data. Even a model that works very well at the beginning will, over time, begin to fail. It’s inevitable to see data drift and model drift. Furthermore, as organizations develop expertise and knowledge on how to apply AI, the use cases, models, and data changes along with those updated applications.
In addition, the global economy and world patterns continue to change in unanticipated ways. This makes long-term planning of any project, let alone complex AI projects, very difficult. In the past two years, retailers likely didn’t anticipate supply chain and labor shocks, and organizations didn’t anticipate the shift in workforce to home-based approaches. As the real world changes and user behavior quickly morphs, the data also changes and models need to change with it. This is why continuous monitoring and iteration of your model is of paramount importance and you must be accounting for these issues with data drift and model drift.


