Data Fingerprinting: An Innovative Lean AI Approach for Startups

Technology startups have immense pressure to embrace machine learning and AI in order to stay relevant in their business. Machine learning, if implemented well, can have a direct impact on a startup’s ability to grow and raise the next round of funding.

However, early stage companies face unique challenges in developing machine learning driven features and products because the quality of solutions is dependent on the quality of the domain-specific data available to train the models. Startups often lack the data specific to the business problem they are trying to solve and generic datasets are not very useful for the unique problems that they are trying to solve. In order to generate the data they need, the product/feature needs to be live. This dilemma either delays or completely prohibits the use of machine learning for many startups.

To overcome this obstacle, startups can embrace a “lean AI” approach by beginning with simple algorithms and incrementally introducing machine learning complexity into the product as they collect more data. For this to happen, there must be a synergy between the data and the algorithms that have the ability to process the data.

Let’s take a closer look at how a lean AI approach can work.

Despite the prevailing belief that data enforces algorithm requirements or data comes before the algorithm, there is a way to use algorithms to define the data requirement. This approach allows developers to gradually work their way to a more sophisticated algorithm for a product feature. The technique of defining the data requirement specific to an algorithm is what we refer to as data fingerprinting.

Data fingerprinting can be thought of as a pattern recognition engine that recognizes certain classes of data in the context of the underlying processing algorithm. By starting with a simple algorithm and limited data, it is possible to incrementally create a more complex algorithm by computationally controlling the nature of the data.

Using the data fingerprinting approach, companies can gate the input to certain classes of data that can be handled by the selected algorithm. If the fingerprint of the input data cannot be recognized, a default logic bypassing the main logic is invoked. It is desirable that the fingerprint generation system be able to transform the input data into a fingerprint irrespective of the type of data. In reality, this is not achievable unless we have data specific processing logic before the data can be fingerprinted.

For example, if the input data is text we need a text processing module that converts text into a standard form.