Dealing with the Lack of Data in Machine Learning
In many projects, I realized that companies have fantastic business+AI ideas but slowly become frustrated when they realize that they don’t have enough data… However, solutions do exist! My goal in this article is to briefly introduce you to some of them (the ones that I used the most) rather than listing all existing solutions.
This problem of data scarcity is really important since data is at the core of any AI projects. The dataset size is often responsible for poor performances in ML projects.
Most of the time, data related issues are the main reason why great AI projects cannot be achieved. In some projects, you come to the conclusion that the data doesn’t exist, that getting it is too hard or takes too much time.
Supervised machine learning models are being used to successfully solve a whole range of business challenges. However, these models are data-hungry and their performance relies heavily on the size of training data available. In many cases, it’s difficult to collect training datasets that are large enough.
The other issue that I noticed is that people tend to underestimate the amount of data necessary for most common business issues. I remember myself struggling to collect training datasets that were large enough. It is even more complicated to collect data when working for a large firm…
How much data do I need?
Well, you need roughly 10 times as many examples as there are degrees of freedom in your model. The more complex the model, the more you are prone to overfitting, but that can be avoided by validation. However, much fewer data can be used based on the use case.
I could also mention the issue of handling missing values. Especially, if the number of missing values in the data are big enough (above 5%).
Once again, dealing with missing values will depend on certain ‘success’ criteria. And these criteria are different for different datasets and (given the same dataset) even for different applications (recognition, segmentation, prediction, classification, etc).
Different solutions exist but it depends on the kind of problem — Time series Analysis, ML, Regression etc.
When it comes to predictive techniques, they shall be used only when missing values are not observed completely at random and the variables were chosen to impute such missing values have some relationship with it, else it could yield imprecise estimates.
In general, different machine learning algorithms can be used to determine the missing values. This works by turning missing features to labels themselves and now using columns without missing values to predict columns with missing values.
Based on my experience, you will be confronted with a lack of data or missing data at some point if you decide to build an AI-powered solution, but fortunately, there are ways to turn that minus into a plus.
As we said earlier, it is impossible to estimate the precise minimum amount of data required for an AI project. Obviously, the very nature of your project will impact a lot your data need. Let’s simplify a lot by saying that text, image, and video usually require more data. However, many other factors are needed to precisely estimate.
In general, small datasets require models that have low complexity (or high bias) to avoid overfitting the model to the data.
Before getting into technical solutions, let’s analyze what we can do to enhance your dataset. It might sound obvious but before getting into AI, please try to get as much data as possible by developing your external and internal tools with data collection in mind. If you know the tasks that machine learning should solve, you can create a data-gathering mechanism in advance.
To initiate ML execution, you could rely on open source data. There are a lot of data available for ML and some companies are ready to give it away.
If you need external data for your project, it can be beneficial to form partnerships with other organizations in order to get relevant data.
