Hadoop is the hot new technology and SQL is the old, tried and tested tool for diving deep into big data, for analysis. This is true, but the number of projects that are putting an SQL front end on Hadoop data stores shows that there is a real need for data querying high level languages in the Hadoop environment. Hadoop MapReduce being a complicated tool for data analysis, developers had come up with Pig and Hive – similar to SQL, which makes it easy to implement Hadoop, without the need for coding in Java, to analyze data. It is important to understand how different these are from each other – this is so that each can be optimally utilized for the right use case.
In the present age of Big Data, a number of querying options are available. While the old giant SQL continues to rein supremacy, organizations’ affinity towards open source programming and querying languages to tame Big Data has created plenty of space for Apache based Pig and Hive. Choosing the right weapon is often half the war won. So, choosing the right platform and language would go a long way in giving you complete control in data extraction, processing and analytics. There is a growing belief that as big data gets bigger; it also needs to get easier. Requirement for faster and easier processing of Big Data is driving the demand for it to get more mainstream.
Talking about Big Data, Apache Pig, Apache Hive and SQL are major options that exist today. All of them have their own advantages in specific situations. Given that the Pig vs Hive, Pig vs SQL and Hive vs SQL debates are never ending, there is hardly a consensus on which is the one-size-fits-all language. Through this article, we present few tips that would help you in choosing the best option to suit a given situation. Before we get into comparisons, we briefly introduce each of them.
Structured Query Language (SQL) has been a programmer’s companion for decades. It was the de-facto solution for extracting data for further processing. Big Data has changed how we visualize and process data. SQL’s demand of storing data in a strict relational database schemas and its declarative nature often deflects focus from the ultimate purpose – to extract data for analysis. For all its popularity, advent of Big Data, challenged SQL’s ability and performance.
SQL programmers required languages that were relatively easy to learn for someone having SQL background and at the same time was –
Originally developed at Yahoo Research in 2006, Pig addressed all these issues and provided better optimization scope and extensibility. Apache Pig also allows developers to follow multiple query approach, which reduces the data scan iterations. It has provisions for a number of nested data types (Maps, Tuples and Bags) and commonly used data operations such as Filters, Ordering and Joins. These advantages have seen Pig being adopted by a large number of users around the globe. Its simplicity has resulted in Yahoo and Twitter resorting to Pig for the majority of their MapReduce operations.
For all its processing power, Pig requires programmers to learn something on top of SQL. It requires learning and mastering something new. Hive statements are remarkably similar to SQL and despite the limitations of Hive Query Language (HQL) in terms of the commands that it understands, it is still very useful. Hive provides an excellent open source implementation of MapReduce. It works well when it comes to processing data stored in a distributed manner, unlike SQL which requires strict adherence to schemas while storing data.
Out of the three approaches to data extracting, processing and analysis, there is no one-size-fits-all approach.
