Which Programming Language Is Best for Big Data?
Nothing is quite so personal for programmers as what language they use. Why a data scientist, engineer, or application developer picks one over the other has as much to do with personal preference and their employers’ IT culture as it does the qualities and characteristics of the language itself. But when it comes to big data, there are some definite patterns that emerge.
The most important factor in choosing a programming language for a big data project is the goal at hand. If the organization is manipulating data, building analytics, and testing out machine learning models, they will probably choose a language that’s best suited for that task. If the organization is looking to operationalize a big data or Internet of Things (IoT) application, there are another set of languages that excel at that.
In the data science exploration and development phase, the most popular language today unquestionably is Python. One big reason for Python’s popularity is the plethora of tools and libraries available to help data scientists explore big data sets. Python was recently ranked the number one language by IEEE Spectrum, where it moved up two spots to beat C, Java, and C++, although Python trails these languages on the TIOBE Index. As a general purpose language, Python is also widely used outside of data science, which only adds to its usefulness.
Another popular data science language is R, which has long been a favorite of mathematicians, statisticians, and hard sciences. The SAS environment from the company of the same name continues to be popular among business analysts, while MathWorks‘ MATLAB is also widely used for the exploration and discovery phase of big data. You also can’t go far in data science without knowing some SQL, which remains a very useful language.
The choice of data science language may also be determined what notebook a data scientist is using. Jupyter is the successor to the iPython notebook, and as such is closely aligned with Python, but it also supports R, Scala, and Julia. The Apache Zeppelin notebook includes Python, Scala, and SparkSQL support.
Programmers will often opt for a different set of languages when it comes to developing production analytics and IoT apps. While they may choose Python or R during the experimental phase of the project, programmers will often rewrite the application and re-implement the machine learning algorithms using entirely different languages.
Java continues to be a very popular choice owing to the large number of Java developers in the world, as well as the fact that some popular frameworks, such as Apache Hadoop, were developed in Java. Scala, which runs inside the Java Virtual Machine (JVM), is also widely used in data science; Apache Spark was written in Scala, and Apache Flink was written in a combination of Java and Scala.
However, for some production applications, developers still favor lower-level languages that run closer to the iron. When speed and latency matter, many developers turn to C and C++ to get them what they want.
MapR Technologies developed its own big data platform, which contained a Hadoop runtime, a NoSQL database, and real-time streaming. But instead of writing its MapR-FS file system in Java, as HDFS was developed, it wrote it in C and C++. As MapR’s Senior Staff Software Engineer Smidth Panchamia explained in this MapR blog post, it’s tough to beat C and C++ for some tasks.
“Native languages like C/C++ provide a tighter control on memory and performance characteristics of the application than languages with automatic memory management,” Panchamia writes. “A well written C++ program that has intimate knowledge of the memory access patterns and the architecture of the machine can run several times faster than a Java program that depends on garbage collection. For these reasons, many enterprise developers with massive scalability and performance requirements tend to use C/C++ in their server applications in comparison to Java.
