Firm foundations are vital for large-scale AI-enabled projects
Successful organisations have learned that systems need to be carefully constructed to provide the feedback which can keep initiatives moving forward.
The clamour of anticipation around new applications for artificial intelligence is as fevered as ever. The problem for me is that expectations are not informed by a robust appreciation of the practical requirements for innovating with AI. As an adviser to businesses on bringing such innovation to market, my advice is simple: to scale rapidly, large-scale AI-enabled projects must be built on firm foundations to allow multidisciplinary development teams to thrive.
Chief among the reasons is that, in engineering terms, developing AI is a complex, non-linear process. Frankly, you can expend a great deal of time and effort with very little progress to show for it. It is the antithesis of agile approaches that deliver incremental advances. Even if you do make solid progress, rapid acceleration at scale will never be guaranteed if the foundations are not fit for purpose. Pushing ahead without a good level of confidence in robust groundwork is risky indeed.
To avoid this risk, three broad foundations need to be set in stone: strategic alignment; data-driven organisation and processes; and a framework for experimenting and iterating in the wild. It is also vital to impose strong organisation and infrastructure when innovating with AI. To get both pillars in place and succeed with an ambitious vision, it is vital to forge strategic alignment amongst end users and key stakeholders as early as possible.
It’s easy to align on the possibilities of AI, but agreeing on the specific use cases and associated engineering complexity and investment may be harder. It’s worthwhile getting this in place early and maintaining a dialogue as ambitions evolve. Capturing this in a multidisciplinary AI strategy from the outset that is updated as the project matures will pay dividends.
If data science and AI teams are going to be able to focus on innovation, we need to organise around them and reduce the friction that can hinder experimentation. We see this through two lenses – model management (which maps quite well to traditional software DevOps practices) and data management (which is newer to some organisations). It seems straightforward, but even reproducing results can be challenging without the right processes in place – as beautifully described by Pete Warden in his blog of a few years ago. Getting this right turns data collection and management into a source of competitive advantage.
