Cutting Big Data Down to Size

Given the mind-boggling scale of big data across all sectors, we need to remember that transporting, storing, and processing this data comes with limits and costs, and is currently on a path that’s unsustainable.

That is the impetus of green ICT—making this technology more energy efficient and thus reducing its carbon footprint. This principle, plus one that would use ICT to “green” other areas, were the central ideas of the IEEE Greening Through ICT Summit, held in October in Paris.

To manage big data, organizations must consider the types of sensors they use to collect this data, how and where the data will be processed, and what kinds of insights they seek. They must take into account what ICT experts refer to as the five V’s: volume, velocity, variety, veracity, and value.

Consider volume in, say, monitoring a patient’s heart rate. The heart-rate waveform itself is of little interest because that pattern has become known over time. Transmitting a constant stream of that data makes no sense—it does not confer useful knowledge. Ideally, a monitor would register only anomalies that threaten the patient’s health and, perhaps, trigger emergency services.

Velocity means the data itself is fast-changing and its usefulness has a time limit. Using the heart-monitoring example, if that data has to be sent upstream for centralized processing and decision-making, the output might be too late to save an at-risk patient.

In terms of variety, think of social media: Different platforms may all convey similar data, but from different angles or with different data qualities. The challenge there might be to reduce redundancies while correlating the different varieties of data in the most efficient way to gain insights.

Another factor to consider is veracity. Big data typically contains errors or unusable points, which require “cleaning it up” or preprocessing it.