When it comes to demanding workloads, few can match big data‘s need for performance, scalability and manageability. Today, big data is a driving force behind a growing number of enterprises using banks of servers, storage and network infrastructure to analyze huge data sets of information to gain or maintain a competitive edge.
Any discussion of big data workloads is almost certainly going to center on Hadoop. If you need to access both structured and unstructured data for deep analytics and sophisticated intelligence, Hadoop is an invaluable tool. In compute-intense environments where server clustering is a standard infrastructure design, such as indexing or understanding customer buying patterns, Hadoop is likely going to be a critical part of the solution set.
By using high-performance algorithms to deduce otherwise hard-to-detect patterns and trends, Hadoop makes big data an essential part of any organization‘s business-critical workload inventory. But Hadoop is highly demanding when it comes to infrastructure requirements. It runs on distributed storage and compute clusters to enable parallel processing of “chunks” of those very large and complex data sets.
Managing TCO in a Hadoop Environment One of the biggest challenges organizations face when implementing and supporting Hadoop workloads is enabling a solution in a cost-effective way, both in terms of capital equipment for the initial deployment and long-term scalability, and for ongoing management of the solution. Hadoop workloads traditionally have required dozens, even hundreds of nodes, which translates into lots and lots of data center racks. If you are using a colocation environment, that means you’ll pay more on a per-node or per-rack basis.
Whether you’re looking at a Hadoop infrastructure node, memory-intensive node or compute-intense node, you can’t just throw more CPUs or disk spindles at the problem to keep up with the demand for these workloads. Not only will the capital equipment expenses spiral out of control, but expanding the underlying infrastructure in such an ad hoc manner is also going to result in the use of much more physical space, as well as higher power and cooling costs.