Four Open Source Data Projects To Watch Now

While open source isn’t the sole source of creativity and progress in big data, it’s a major driver in the space. Market-shaking tech like Kafka, Spark, Hadoop, and MongoDB all began as obscure open source projects backed by enthusiasic developers. Which open source project will be the next breakout star?

Many open soruce projects are organized under the Apache Software Foundation, but not all. Here are four recently founded upstream big data projects – two hosted by the ASF and two that aren’t —  that could find their way into your quiver of big data analytics tools.

Apache Fluo is an open source implementation of Percolator, the Google software developed to speed the incremental processing of updates to the Google search index. The Hadoop-based software runs atop Apache Accumulo, the open source key-value database based on Google’s Bigtable technology, and “makes it possible to update the results of a large-scale computation, index, or analytic as new data is discovered,” the project says.

Fluo is said to provide better performance for incremental updates on large data sets compared to batch-oriented Spark and MapReduce engines. It also eliminates the need to reprocess entire data sets when adding new data to the existing set, a common concern in data lakes.

Fluo graduated into a Top-Level Project at the ASF yesterday. “Apache Fluo is a very clever piece of software, elegantly supplementing Apache Accumulo’s ability to store and maintain very large indexes,” said Christopher Tubbs, ASF Member and Committer on Apache Accumulo and Apache Fluo. “Its support of transactions enables Accumulo to solve a whole new set of big data problems, and its observer framework makes designing ingest workflows fun.”

The project includes a core API that supports simple, cross-node transactional updates, and a Recipes API that supports complex transactional updates. It’s managed at fluo.apache.org.

Apache Lens is an OLAP technology for Hadoop that provides a unified model and execution engine for querying data that resides in multiple locations, including Apache Hive and traditional data warehouses, such as HPE Vertica or Teradata, ostensibly.

“Lens aims to cut the data analytics silos by providing a single view of data across multiple tiered data stores and optimal execution environment for the analytical query,” it says on the project’s GitHub site.

According to Apache Lens Vice President Amareshwari Sriramadasu, the technology ” solves a very critical problem in big data analytics space with respect to end users. It enables business users, analysts, data scientists, developers and other users to do complex analysis with ease, without knowing the underlying data layout.”

The software was initially devloped at InMobi, the mobile ad network company, and has been an open source projecdt at GitHub since 2013. It became a Top-Level Project at the ASF two years ago, and is managed at github.com/apache/lens.

Deck.GL is a compelling data visualization framework developed by Uber and released as open source in 2016.