Small data observability can provide the real-time insights that matter, empowering companies to maximize the uptime of complex and growing infrastructure.
Capturing data for systems observability is a growing priority for organizations. Not only is it important for maximizing IT efficiency, but it is also key to identifying and responding to issues with security and performance. When it comes to SRE, IT operations, and other critical teams, capturing and analyzing the right data to draw conclusions and make informed decisions makes a huge difference in their speed and effectiveness. As more data is generated at the edge, it is pushing teams to reevaluate their observability strategy.
The most common approach companies take involves moving raw edge data to a central repository and analyzing it in batches – a “big data” observability strategy. However, this has become infeasible – or at the very least, very expensive – at scale. The world produced an estimated 79 zettabytes of data in 2021, leaving many companies overwhelmed by the expenses of storage and analysis in the form of software-as-a-service (SaaS) contracts and cloud expenses.
The nature of the edge, and data on the edge, also presents unique concerns. Companies face the challenge of configuring, deploying, and maintaining agents in thousands of locations at once in order to extract all the data being produced. Said locations might simply be altogether incompatible with edge visibility solutions that depend on on-premises hardware or VMs for observability. There’s simply no room for these solutions on an IoT device or lean branch location server.
There is an alternative approach to enabling observability on the edge – push analysis out to where the events happen and pair it with the ability to dynamically control what is analyzed in real time. We call this “dynamic edge observability,” or what you may call a “small data” strategy. This approach empowers teams to conduct and retrieve analyzed data in real time, scaling commensurately with the raw data edge infrastructure generates. This offers unrivaled specificity and flexibility, speeding up incident response while keeping costs stable even as companies analyze more data.
Still, big and small data aren’t mutually exclusive: big data can provide the context needed to better leverage the hyper-specific small data approach. Let’s take a closer look at how big and small data approaches to observability compare, how small data can dynamically generate insights, and how the two approaches can work in tandem.
The big data approach to observability involves bringing raw telemetry from the edge to a central repository where a SaaS or cloud provider can perform analysis on it to generate insights. When faced with the question of how much data should be collected, teams often default to thinking “as much as possible” in the hopes of having what they need when the time comes to ask questions.
Unfortunately, this strategy has significant drawbacks.
