It has been found in industrial circles that enterprises are struggling to make sense of all the information they have. In the recent past, their focus on data has grown manifold and data analytics has become more effective since enterprises have access to Big Data.
Big Data is a collection of different data management applications that support multiple analytics uses. Analytics without the ability to manage large data is not effective, and so, enterprises explore to leverage Big Data technology such as Hadoop.
Hadoop is an open-source framework for storing data in distributed computers and processing this data in parallel on clusters of commodity hardware (i.e. computers). Analytics can be relevant if there is data to work on and, in today’s world, data itself is huge.
This data, stored in various database management systems can go up to several petabytes. It is a moving target and no threshold has been defined as such.
The architecture of Big Data consists of several racks of storage nodes with many components. The architecture may vary from user to user depending on requirements. Commercial interest in this area can be gauged by the fact that private equity and various venture capital funds are investing in Big Data related initiatives.
Big Data investments are expected to grow at CAGR of 17 percent over the next three years, eventually accounting for $76 billion by 2021.
Another interesting development is that of acquisition of pure-play Big Data start-ups. The competition in the area among Big ITES enterprises is growing and that is the reason for the run for new start-ups and acquisitions.
At this stage, hardware and infrastructure sales in Big Data accounts for nearly 70 percent of total investment and that is due to the requirements for large scale servers, routers, gateways and variety of network components.
In the following we outline important developments in Big Data, including technologies in data storage and processing, which have had a remarkable imprint on Big Data technology decision-making. Our observations are indicative in nature and shed light on the trajectory these developments are taking, but are not meant to be comprehensive and do not encompass all the facets of Big Data technologies.
As data keeps on growing so is the need to find cost effective solutions to store this data and use it when necessary. In this context, the concept of data lakes has become practical and useful solution.
A data lake is storage that holds raw data in its native format and uses a flat architecture to store data. Going forward, the share of software vendors will increase who would build products to read raw data from data lakes in a more cost effective and efficient form.
Traditionally, SQL (Structured Query Language) is a query language used by Relational database Management Systems (RDBMS). Relational databases rely on tables & columns to store and extract data.
In recent developments, NoSQL database management systems have evolved where NoSQL databases do not rely on these structures and use more flexible data models. NoSQL is useful for storing unstructured data (images, video & audio recordings, text), which is increasing more rapidly than structured (numeric) data and does not fit the relational form of RDBMS.
Common types of unstructured data like weblog files, chat files, cellphone messages, data from the Internet of Things (IoT) devices and video and images.
Let us look at a specific example where the capabilities of NoSQL database are used. A customer profile today has a variety of information that includes data from different touch points of the customer. Customer data could include images, recordings, text data, locations, web browsing history, customer links (friends & family) and customer service data.
This data is crucial to the enterprise in order to more effectively serve customer. NoSQL databases can provide much faster data extraction and loading to specific portions of customer data.
