The Secrets to Scaling AI: It’s About People and Software!
Every business function struggles with scale, including AI and data science. Architects have addressed technical scale to increase productivity — blazing fast connections, big data architectures, faster chips, distributed computing, massively parallel processing, and all sorts of optimized hardware stacks for AI.
Bigger, faster systems are only part of the solution to scaling AI. One data scientistworking on a massively parallel system (no matter how elastic or scalable) is still only one data scientist. So for AI to thrive in business, we must change how we think about “scaling” data science work.
So what’s the key to unlocking these constraints? AI needs to be scaled in three ways, technically, organizationally, and operationally. This blog will focus on the second pillar, scaling organizationally with a software-centric approach.
If we’ve learned anything over the past decades of automation, people need integrated software platforms to do their jobs efficiently. But most AI platforms were designed only for expertdata scientists. But expert data scientists have a super-set of skills that even other technical experts, like engineers in the business, don’t have.
Traditional data science platforms and services celebrate the complexities of data science, requiring deep coding and technical skills and leaving most of us on the sidelines. An essential element of scaling is software architecture.
Software applications (now delivered as-a-service in the cloud) support “teams” and have been the key to unlocking the value of almost every business function you can think of — from accounting, to marketing, to operations. Software simplifies complex workflows, tooling environments, and processes while enabling teams.
Software helped systemize these functions and create departments that scaled beyond the capabilities of their most expert accountant, their most prolific marketer, or the lone genius in operations. Software architecture enabled new users, created a common platform, and connected “super users” (expert data scientists in our context) with business users (citizen data scientists and business analysts).
According to Deloitte, data science is on the same maturity path:
Dataiku, the platform for Everyday AI, exemplifies this approach. It’s a software-centric approach to creating scale for data science organizations. Whether running on your chosen infrastructure or delivered as-a-service, the software platform combines the ability to scale data science teams and leverage elastically scalable infrastructure. It enables both expert data scientists and non-experts on a common, unified platform.
A modern AI software architecture needs six key design elements to allow AI to scale:
What beats a super-genius Ph.D. data scientist working on a screaming-fast, massively parallel infrastructure? A “team” of data scientists working with skilled data engineers, business domain experts, business analysts, AI developers, and operations as a team on a common, connected platform.
Dataiku connects people to teams, projects, tools, data, work, and insights that help them do their jobs. Data scientists must be connected to data engineers, operations, business analysts, domain experts, and others. And they need to be on the same page around projects, as well as to communicate about and share those projects from inception to production. And they all need a single view of any given project.
Data science teams will have a broad range of skill sets. An AI software platform needs to support expert data scientists who prefer working in code and non-technical users (i.e., business analysts) who choose a visual interface. Dataiku enables both modes.
In Dataiku, coders have all the tools they need to customize projects working programmatically. Coding in any language is fully enabled for any IDE or programming language with Dataiku code studios.
