Robust Algorithms for Machine Learning

Machine learning is often held out as a magical solution to hard problems that will absolve us mere humans from ever having to actually learn anything. But in reality, for data scientists and machine learning engineers, there are a lot of problems that are much more difficult to deal with than simple object recognition in images or board games with finite rule sets.

For the majority of problems, it pays to have a variety of approaches to help you reduce the noise and anomalies so you can focus on something more tractable. One approach is to design more robust algorithms where the testing error is consistent with the training error, or the performance is stable after adding noise to the dataset.

The idea of any traditional (non-Bayesian) statistical test is the same: We compute a number (called a “statistic“) from the data and use the known distribution of that number to answer the question, “What are the odds of this happening by chance?” That number is the p-value.

The problem with this approach is that the “known distribution” of that number depends on the distribution of the data. This dependency can be mild — as in the case of Student’s t-test or the F-test — or it can be so severe and make the value essentially meaningless for statistical purposes. Pearson’s “r” (which appears as r-squared in linear regression problems) falls into the latter category, as it is so sensitive to the underlying distributions of data that it cannot, in most practical cases, be turned into a meaningful p-value, and is therefore almost useless even by the fairly relaxed standards of traditional statistical analysis. For example, using “r” as a measure of similarity in the registration of low-contrast images can produce cases where “close to unity” means 0.998 and “far from unity” means 0.98, and there’s no way to compute a p-value due to the extremely non-Gaussian distributions of pixel values involved.

Statistics of this kind are sometimes called “parametric” statistics due to their dependency on the parameters of the underlying distributions. Student’s t-test, for example, depends on the distributions being compared having the same variance.

Robust statistics are also called nonparametric precisely because the underlying data can have almost any distribution and they will still produce a number that can be associated with a p-value.

The trick is to find a property of the data that does not depend on the details of the underlying distribution. In particular, converting cardinal data value to ordinals (ranks) allows us to ask some very robust questions.

Take, for example, the Mann-Whitney U test.