How to fight the Hydra of large-scale data challenges
Dealing with scale is a multi-dimensional challenge and, if you’re not careful, it may feel like you’re battling a multi-headed monster.
Just when you think you’ve solved scale by having a way to store large amounts of data, another problem pops up. Solving issues created by large-scale systems is like facing the mythical creature Hydra – when you cut off one of its heads, two more grow back, ready to consume you. How do you survive?
You’ll find a hint in the myth: deal with allthe monster’s heads, not just one, to defeat it. You cannot win by solving a single challenge of scale: you need to be aware of different aspects of the problem, some of which may be hidden.
You should be able to meet your SLAs, in an affordable way, at your current scale and as your system grows. For this to be practical, you must do it without having to scale up your IT team to match the growing quantity of data.
How can you do this? Start with the idea that “big” isn’t just about the number of bytes.
There’s more to scale than the number of bytes
Scale includes large amounts of data but also involves other things, such as data diversity. Traditionally, people mainly had files and databases. Now data includes images, media, events, and so much more. Data diversity is just one issue you face with large-scale projects. Either you conquer the various aspects of scale, or the problems that ensue will be large.
Here are four key dimensions of scale, each of which must be won over.
How big is big data? People work with amounts of data today that just a few years ago would have been considered unusual. Consider it big when the size of data itself becomes a technical challenge. People who work with less than 50 terabytes may not feel the challenge of scale. The HPE Ezmeral team routinely works with businesses that have data ranging from 50 terabytes to 500 petabytes and beyond, so their data infrastructure needs to handle scale easily.
One way to conquer large data size is to not make it bigger than it needs to be. Limitations imposed by some data technologies lead people to make unnecessary copies of huge data sets. This happens in part because their infrastructure lacks open and flexible data accessmethods needed to accommodate different analytics or machine learning tools and languages.
Data also gets copied unnecessarily to deal with “noisy neighbors” – competing applications that may create hot spots or congestion. Data infrastructure should have platform-level capabilities that automatically make local copies of small subsets of data to avoid hot spots. In addition, your data infrastructure needs fully distributed metadata to avoid congestion when multiple applications access large data sets.
Finally, when you have a lot of data, you need a convenient way to find it. Data infrastructure with familiar file and directory access and a global namespace can improve the efficiency of your team because references to data can remain stable.
The lesson is this: Conquer data size while meeting SLAs with the help of your data infrastructure. It should provide platform-level capabilities for secure, affordable large-scale data storage plus flexible data access and management. This lets you meet current and future challenges of scale. Businesses shouldn’t have to re-architect their system as data grows.
2. Don’t be slayed by the number of objects
Another challenge of scale is a large number of objects.
