Machine Learning that Automates Data Management Tasks and Processes
Machine learning is not just for predictive analytics. It can also be embedded within tools to automate data management development and optimize execution.
Today, most efforts with machine learning (ML)are to support predictive analytics, especially when the analytics parse vast amounts of diverse big data. This is an important practice, and it will continue to grow and mature.
However, a few cutting-edge vendors and open-source projects are embedding ML-driven intelligence into data management (DM) tools. Embedded within DM tools, ML algorithms and models typically address three broad goals:
Machine learning is high value in these contexts because it increases developer productivity, makes advanced functions doable by lightly technical users, and elevates system performance with minimal administrator involvement. Due to these compelling benefits, TDWI expects to see — in a few years — most DM functions automated or optimized via ML and other approaches (e.g., rules engines). Here are a few examples.
Data cataloging. Modern tools can catalog and categorize data automatically via machine learning algorithms and models as well as via old-school business rules and application logic. Cataloging can apply to data sources, datasets, tables, or even individual columns and fields. A single data element can be categorized by its domain, compliance risk, quality level, source, lineage, and so on, as the user organization requires. Cataloging each data element multiple ways enriches user searches and queries of the catalog, and it enables richer cross-category analytics correlations.
Data domains. ML algorithms and other tool logic can recognize and catalog data sources and structures that are of particular domains. This helps users who will browse or search the catalog for domains of high interest, such as the customer, product, and financial domains. Advanced algorithms can even detect domains and domain relationships across datasets. ML algorithms can also recognize and catalog data elements that are potentially sensitive in terms of privacy and compliance.
Data lineage.
