How Spark and Hadoop Are Advancing Cancer Research
The combination of Spark and Hadoop has supercharged big data analysis across many industries and use cases by lowering the barrier of entry to advanced analytics and thereby enabling data scientists to create data-driven products that weren’t previously possible. But one area where Spark and Hadoop are having an especially strong impact revolves around cancer research.
Cancer killed about 590,000 Americans last year, according to the Centers for Disease Control. That makes it the second leading causes of death in the United States, behind only heart disease, which killed 615,000. Forty-five years after President Nixon signed the National Cancer Act of 1971, which effectively was a declaration of war on cancer, progress against the collection of diseases has more or less stalled.
While an outright victory against cancer remains unlikely, there are signs that the momentum may be starting to shift in our favor. President Obama’s Precision Medicine Initiative (PMI) and Vice President Biden’s “Moonshot” program are shining the spotlight (and funneling funding) to promising approaches to fighting cancer. And a lot of the new approaches involve big data, in one way or another.
As two of the most powerful and popular products in the big data space, it’s no wonder that cancer researchers are gravitating toward Apache Spark and Apache Hadoop to tackle difficult compute-intensive tasks. Here are two examples of how the combination of Hadoop and Spark are helping to reshape the landscape in the field of cancer research.
One big supporter of the new parallel computing architectures like Hadoop and Spark is Jay Etchings, Arizona State University’s director of operations for research computing and a senior HPC architect. Etchings is credited with installing the first production Hadoop cluster at the university, and today the 44-node Hortonworks (NASDAQ: HDP) cluster is being used for a variety of projects, including cancer research.
Etchings says the Red Hat model of commercial open source software has been a boon for computational scientists and researchers, who previously had to hand-code a lot of their own HPC tools to get stuff to work. Etchings had first-hand knowledge of this while working in the pharmaceutical industry.
“For years I spent time tinkering with things like PLINK/SEQ and other tools and getting the appropriate libraries installed and getting stuff to work so I could actually do the science I wanted to do,” he tells Datanami. “The academic code is unreproducible, unbuildable, undocumented, unmaintained, backward incompatible [very bad] code. That’s really what it is.”
The rise of the Apache Hadoop/big data ecosystem has changed things dramatically for folks like Etchings, who are constantly looking for ways to help university researchers. Etchings appears quite happy with ASU’s Hadoop cluster, which leverages Spark, Hbase, Hive, Accumulo, and visualization tools from Tableau (NYSE: DATA).
“The architecture in itself is good because it gives us the robust scalability, the same sort of parallelization that we’d get in HPC, but without converting to MPI or writing in MPI. We don’t have the difficultly we had in HPC to scale horizontally,” he says. “You put something on it, and you run a job, and it’s already running across maple nodes. Even within Spark, you align a computation, you split up into RDDs, and it goes ahead and runs the mathematic computation.”
One of ASU’s projects involves analyzing the genetic diversity involved in Glioblastoma multiforme (GBM), a particularly lethal form of brain tumor. While the prognosis for those afflicted with GBM is generally poor, doctors have noticed there can be significant differences in survivability rates, and they haven’t been able to figure out why. Now researchers are beginning to discover what’s going on under the surface.
