How to Scale an AI Platform: It’s Not Just About “Speeds and Feeds”
There are many ways to achieve scale in AI and machine learning (ML) — scale up, scale out, elastic scale. But taking a more granular approach to scaling your AI/ML projects can pay dividends. The best way to understand scale for an AI and ML platform is to look at each step in the lifecycle of a project. Each stage from design to production has different types of workloads and there are different ways of scaling to meet user needs.
While highly scalable infrastructure is critical, the secret to scaling AI lies in the design of your AI platform and how it scales at each of the major steps in the AI/ML lifecycle, notably Prepare, Build, Deploy and Monitor (which we’ll discuss in more detail below). For IT/cloud architects and analytics leaders, scaling at the right time in the right way is critical to enabling your data science teams.
In Dataiku, each of these stages has dedicated environments that help scale at each step in the life cycle. Scaling in this way enables teams — business analysts, data engineers, data scientists, ML engineers, and IT operations — with a highly performant elastic AI stack for their individual stage in the process and also optimizes the entire AI/ML lifecycle.
The data preparation phase in AI/ML includes things like data acquisition, discovery, visualization, shaping the data, transformations, feature selection, pre-processing, and exploratory analysis. It is the stage at which we prepare data for modeling.
Scalability in this context begins with the ability to work with larger datasets and more types of datasets. With access to massive amounts of data in cloud data warehouses and data lakes, maintaining performance becomes even more challenging. Dataiku has pre-built connections to all the major cloud petabyte and exabyte storage, including databases, cloud data warehouses, and blob storage, but takes an innovative approach to preparing these massive data volumes for modeling, whether structured, semi-structured, or unstructured.
In the data preparation step, performance is critical when making multiple transformations on large volumes of data. Dataiku uses pushdown execution for data preparation tasks and users can choose the most effective engine based on the data sources and recipes (local, in-database, or Spark). Pushdown workloads can be executed efficiently in elastic cloud compute clusters or in-database. But, more importantly, Dataiku maintains accessibility to your data, no matter the size of your dataset, so you view and work with it through the same interface.
Pushdown helps optimize the runtime for Dataiku projects. When running on Spark or a SQL database, a long chain of recipes (data preparation tasks) can be executed without building intermediate output. This helps teams avoid cycles of reading and writing data at each intermediate step and reduces or eliminates data movement over the network. Business analysts, data engineers, and others involved in the data preparation phase benefit from both the speed of processing but also from a host of data preparation tools and techniques in the platform.
Scale takes on another dimension when we move into the experimentation and model building phase of an AI/ML project, which is inherently iterative. It is essential to reduce the time and expense associated with model experimentation and training by leveraging scalable infrastructure for these tasks with both flexible AutoML and training of custom models. The requirements to scale training mostly rely on computational power and model parallelization/distributed compute. For data scientists, the ability to experiment and iterate is crucial, but has to be managed with cost in mind so speed and scalability are also critical.
Dataiku contains a powerful AutoML engine that allows you to get highly optimized models with minimal intervention.
