Top Big Data Processing Frameworks

A discussion of 5 Big Data processing frameworks: Hadoop, Spark, Flink, Storm, and Samza. An overview of each is given and comparative insights are provided, along with links to external resources on particular related topics.

With the modern world’s unrelenting deluge of data, settling on the exact sizes which make data “big” is somewhat futile, with practical processing needs trumping the imposition of theoretical bounds. Like the term Artificial Intelligence, Big Data is a moving target; just as the expectations of AI of decades ago have largely been met and are no longer referred to as AI, today’s Big Data is tomorrow’s “that’s cute,” owing to the exponential growth in the data that we, as a society, are creating, keeping, and wanting to process. As such, traditional data processing tools which do not scale to big data will eventually become obsolete.

So the question is, what are we doing with this data? The answer, of course, is very context-dependent. But everyone is processing Big Data, and it turns out that this processing can be abstracted to a degree that can be dealt with by all sorts of Big Data processing frameworks. A few of these frameworks are very well-known (Hadoop and Spark, I’m looking at you!), while others are more niche in their usage, but have still managed to carve out respectable market shares and reputations.

We will take a look at 5 of the top open source Big Data processing frameworks being used today. Of course, these aren’t the only ones in use, but hopefully they are considered to be a small representative sample of what is available, and a brief overview of what can be accomplished with the selected tools.

First up is the all-time classic, and one of the top frameworks in use today. So prevalent is it, that it has almost become synonymous with Big Data. But you already know about Hadoop, and MapReduce, and its ecosystem of tools and technologies including Pig, and Hive, and Flume, and HDFS. And all the others. Hadoop was first out of the gate, and enjoyed (and still does enjoy) widespread adoption in industry.

So why would you still use Hadoop, given all of the other options out there today? Despite the fact that Hadoop processes often complex Big Data, and has a slew of tools that follow it around like an entourage, Hadoop (and its underlying MapReduce) is actually quite simple. If your data can be processed in batch, and split into smaller processing jobs, spread across a cluster, and their efforts recombined, all in a logical manner, Hadoop will probably work just fine for you.

A number of tools in the Hadoop ecosystem are useful far beyond supporting the original MapReduce algorithm that Hadoop started as. Of particular note, and of a foreshadowing nature, is YARN, the resource management layer for the Apache Hadoop ecosystem. It can be used by systems beyond Hadoop, including Apache Spark. Here is an in-depth article on cluster and YARN basics.

Spark is the heir apparent to the Big Data processing kingdom. Spark and Hadoop are often contrasted as an “either/or” choice, but that isn’t really the case. The Hadoop ecosystem can accommodate the Spark processing engine in place of MapReduce, leading to all sorts of different environment make-ups that may include a mix of tools and technologies from both ecosystems. As one specific example of this interplay, Big Data powerhouse Cloudera is now replacing MapReduce with Spark as the default processing engine in all of its Hadoop implementations moving forward. As another example, Spark does not include its own distributed storage layer, and as such it may take advantage of Hadoop‘s distributed filesystem (HDFS), among other technologies unrelated to Hadoop (such as Mesos).

Spark differs from Hadoop and the MapReduce paradigm in that it works in-memory, speeding up processing times. Spark also circumvents the imposed linear dataflow of Hadoop‘s default MapReduce engine, allowing for a more flexible pipeline construction.