Microsoft taps into Apache Spark to drive its Big Data & analytics services

Microsoft is making what it claims is an “extensive commitment” to the Apache Spark Big Data processing engine, launching several new offerings out of preview and into general release.

The move is the latest in Microsoft’s embrace of open-source technologies, a trend that only emerged in the last couple of years after CEO Satya Nadella took over the company. Microsoft made its announcements at the Spark Summit 2016 developer’s conference in San Francisco yesterday.

Microsoft launched Spark for Azure HDInsight in preview last year, promising it would be the “best environment to run Apache Spark” with a managed service in the cloud. Now, Spark for Azure HDInsight has been made generally available, together with a fully managed Spark service from Hortonworks Inc. that’s been “hardened for the enterprise and made simpler for you to use,” the company said.

In addition, Microsoft announced the general availability of R Server for HDInsight later this summer. Currently available in preview mode, R Server for HDInsight include Spark integration for both the on-premises and cloud versions. The “R” refers to the R programming language that’s used for statistical computing and predictive analytics. Microsoft is one of the leading proponents of R following its acquisition of Revolution Analytics in 2015.