Data transformation: The missing ingredient from edge computing strategies

It’s easy to get excited about the Things in the Internet of Things (IoT). Tiny computers, factory robots, devices with 3D cameras, devices with GPS units — everything from telemetry units on long-haul freight trucks to gumball-size sensors reporting from atop the cooling towers of power plants. The world of network-connected devices, whether specialized or general purpose, is more varied and creative than ever.

The innovation taking place with this hardware is truly impressive. But to take full advantage of the hardware, it’s important to pay just as much attention to the software. Edge computing is going to quintuple the amount of data enterprises collect from devices in just the next three years. quintuple the amount of data that enterprises collect from devices in the next three years alone. How to manage that data and make the most of it — well, that’s a pretty edgy job.

The success of many organizations’ IoT initiatives will hinge on data, data integration and data management. Fail to architect data integrations and transformations properly, and IoT systems will have trouble scaling and delivering the profound business transformations their vendors have promised. Mishandle the data of the thousands of IoT devices in use, and the business will underperform.

Here’s an example to illustrate my point.

Let’s say you’re using IoT devices to monitor production equipment on a factory floor and have thousands of these devices active across the entire assembly line. If any of these devices report an error, you need to analyze the error right away. Worst case, you might want to stop production until the error is corrected to prevent damage to the material in production, or even to the factory equipment. Finally, let’s imagine that these thousands of devices are reporting status every 30 seconds.

You have a couple choices here:

From the perspective of IT performance, the second scenario makes a lot more sense (even though the first is even being championed by some major cloud software vendors). There’s nothing to be gained by sending all status OK messages to the cloud, they simply consume bandwidth. The second scenario dramatically reduces network traffic and handles everything else at the network edge.

Now let’s make our example a little more complicated — and here I’m going to draw on a real-world use case.

It is very typical for IoT scenarios to entail the collection of all, or most, of the data from a landscape of devices for the purposes of storing history and mining for insights via analytics tooling. It is also very common to have to transform that data, which is often done by the way of stored procedures since the common wisdom is to transform at the database.