Will Apache Spark Finally Advance Genomic Data Analysis?

For most, the thought of big data is often accompanied with that of big companies like Twitter, YouTube or Google. The last decade or so has shown massive advancements in the capabilities of storage and data processing like Apache Hadoop and as these big data technologies have grown, so have their uses. More and more industries are using big data to gain perspective in their markets and access their customers more efficiently through holding personal data at scales never before imagined. Companies can keep track of their employees and customers better, offering a more personalized approach to shopping and also keeping their workers up to snuff (think current licenses and things like ACLS certification).
Currently, YouTube and Twitter are the leaders in the amount of data that is collected and analyzed each year. YouTube has about 1-2 exabytes of video data uploaded per year. Twitter has estimated growth equalling 1.36 petabytes of data per year by 2025. But the need for big data developments has never been greater than with genomics. The first human genome cost nearly $3 billion to sequence and took about a decade to do so. Shortly thereafter, the available methods to sequence improved and genomic data analysis research skyrocketed with a lot of the work aimed at finding new and better ways to treat diseases. However, all of this new growth has created a data crunch with estimates that will far out grow both Twitter and YouTube. By 2025, we could reach one zettabase of sequence per year with the need of 2 – 40 exabytes of storage capacity for genomic data.
Before, research teams were using Hadoop clusters to run their analyses, attaching on other scripting and analysis platforms. Unfortunately, processing times were slow and compatibilities across platforms such as Hadoop and Apache Pig were problematic. Other tools like the 1000 Genomes dataset can cope with up to a few thousand genomes, but are unable to handle datasets any larger than that.


