How artificial intelligence and machine learning are changing the development landscape [Q&A]

It’s an increasingly rare application these days that doesn’t claim to incorporate some form of artificial intelligence or machine learning capability.

But while this may be great from a marketing standpoint it does pose a challenge for developers. We spoke to Luis Ceze, CEO and co-founder of OctoML, to find out more.

BN: We know developing AI/ML has nuances that differ from traditional software development? What are some of the challenges developers run into?

LC: AI is the new ‘brains’ of the app — much like databases before it. But unlike databases (and virtually all other types of software) AI is largely inaccessible to application developers and operations teams. Why? Because the process of actually getting the machine learning models that power AI into these smart applications is unnecessarily complex and cumbersome. Not only that, but models are dependent on specific hardware and software infrastructure to perform well enough to be viable.

What we have today is a fundamental disconnect between the process of creating these models, and actually getting them to production. As a case in point, the average time-to-production for an ML model is 12 weeks, and 47 percent of models are dead on arrival — which is unsustainable if AI is going to make a dent on the world.

The crux of the issue is that ML models are treated as this bespoke thing that requires specialized workflows and tools — when in the end, after being created, they are just code and data, and should be treated as ‘regular’ software modules.

The deployment requirements of ML are reminiscent of the ‘matrix from hell’ that Docker helped exorcise from software development. What’s even worse this time around is the ML software stack is fully-dependent on the hardware where it runs. Then you must hand-tune models to meet performance SLAs for the application. We’re now living in the era of the ‘tensor from hell’.

Let me break down just how painful the process is today. Almost every model created ends up with its own unique pipeline. These are generally hand-crafted, with very little variability, and even the best data scientists and AI practitioners often get it wrong. They’re very fragile. One change in the environment, training framework, software library, or integration stack, and the whole thing needs to be debugged, or worse, completely rebuilt.

This is the major reason why the hand-off from the data scientists to the app developers and ops teams is so messy — it’s plagued with trial-and-error. In order to break this complex cycle and accelerate AI application development, machine learning must align with DevOps workflows and best practices.