Visualization Enables High Dimensional Analytics

High dimensional data is our greatest asset in learning from data sets with hundreds, even thousands, of variables.

Data visualization is undergoing a revolution, making complex data sets easier to understand and helping both experienced and inexperienced analysts form better conclusions and takeaways from those numbers.

A notable side effect of increased capabilities for data visualization is a push toward more complex modes of data collection and processing; if we’re able to understand complex data sets without needing substantial training or experience, we can apply those data processing standards to more areas.

In the era of big data, we’ve been able to collect and store more data points than ever before. Rather than relying on simple bits of information about key demographics and behaviors, we have access to hundreds, and sometimes thousands of variables related to a given problem or outcome. For example, in medical research fields, characteristics include genetic predispositions, lifestyle factors, and demographic information may all play a role in whether a patient develops a condition (and how they respond to treatment). Each of these hundreds of variables may interact with any of the other variables, making it impossible to do a simple correlational analysis in variable pairs or triplets.

It’s difficult to imagine anything in more than three dimensions, but for computers, it’s relatively easy. In physics and computer science, mathematical models can be used to make calculations in higher dimensions, sometimes hundreds of dimensions, allowing us to crunch the numbers and uncover patterns. There’s only one significant obstacle to making this practical: visualizing the results.

The simplest model of data visualization is also the first one most of us are introduced to: the bar graph, in which one set of variables is plotted on the horizontal x-axis, and another is plotted on the vertical y-axis. This is highly effective, but only extends to two dimensions of data.

Researchers have developed multiple techniques to push the limits of what we can visualize, and most of them focus on reducing the number of presentable dimensions, in some way, to three or four.