What Is Overfitting in Machine Learning, and How Can It Be Prevented?
In general, overfitting refers to the use of a data set that is too closely aligned to a specific training model, leading to challenges in practice in which the model does not properly account for a real-world variance.
In an explanation on the IBM Cloud website, the company says the problem can emerge when the data model becomes complex enough that it begins to overemphasize irrelevant information, or “noise,” in the data set.
“When the model memorizes the noise and fits too closely to the training set, the model becomes ‘overfitted,’ and it is unable to generalize well to new data,” the company writes. “If a model cannot generalize well to new data, then it will not be able to perform the classification or prediction tasks that it was intended for.”
So, because of its contours favoring the data that it was trained against, the data model is more likely to produce false positives or false negatives when used in the real world.
In some ways, overfitting stems from issues with how the original data model was built, creating gaps in the machine’s understanding.
This can happen for many reasons — importantly, that a model was built for specific outcomes rather than slightly more generalized ones. (There is also a threat of the opposite problem, underfitting, which happens when the data model isn’t mature enough, creating false positives or false negatives.)
Overfitting can introduce inefficiency into the business, adding costs. For example, overfitting can lead to issues in detecting security threats to internal platforms, allowing risks to enter a network undetected. When used in data forecasts, it can create a misunderstanding of how big the need for a product is, leading to problems with how that demand is managed within the supply chain.
In some cases, overfitting can represent a form of algorithmic bias, in which errors in the data model create negative outcomes for the end user — for example, if people are more likely to be denied for a loan or credit based on a predetermined level of risk that doesn’t account for their specific circumstances.
Challenges in attempting to build data models ethically reflect the importance of taking steps to avoid overfitting when bias or discrimination is a concern.
Cassie Kozyrkov, the chief decision scientist at Google, said in a 2019 presentation that a key element in battling algorithmic bias caused by methods such as overfitting is to test heavily against the available data.
“Computers have really good memory,” Kozyrkov said, according to VentureBeat. “So the way you actually test them is that you give them real stuff that’s new, that they couldn’t have memorized, that’s relevant to your problem. And if it works then, then it works.
