How to build an all-purpose big data pipeline architecture
The quickest and often most efficient way to move large volumes of anything from point A to point B is with some sort of pipeline. Whether associated with lanes on a superhighway or major arteries in the human body, pipelines can rapidly advance objects and enable them to easily diverge and perform tasks along the route. The same principle applies to a big data pipeline.
To put the term big data into context, when data and the frequency at which it’s created are small, an email with an attached document will suffice for transferring it and a hard drive will suffice for storing it, said David Schaub, a big data engineer at Shell. When the data is small and the frequency is high, it makes sense to automate sending documents or storing them with a simple out-of-box tool. When it strains the limits of out-of-box tools for either transfer or storage, the data is considered “big.”
“This necessitates a tool that takes more configuration than normal,” Schaub explained. “A big data pipeline is tooling set up to control flow of such data, typically end to end, from point of generation to store.”
One of the more common reasons for moving data is that it’s often generated or captured in a transactional database, which is not ideal for running analytics, said Vinay Narayana, head of big data engineering at Wayfair. To be most useful, this data often needs to be moved to a data warehouse, data lake or Hadoop file system (HDFS) — or from one data store to another in batch or real time. “The big data pipeline enables the handling of data flow from the source to the destinations, while calculations and transformations are done en route,” noted Serge Vilvovsky, founder and CEO of cloud data lake security provider AltaStata and a member of the MIT Sloan cybersecurity consortium. “Using the pipelines, organizations can convert the data into competitive advantage for immediate or future decision-making.” Big data architecture and engineering can be complex. Therefore, software engineers with some experience in dealing with large amounts of data are generally involved in building a data pipeline. Some data, such as free text, may require data scientists.
A big data pipeline enables an organization to move and consolidate data from various sources to gain a unique perspective on what trends that data can reveal, said Eugene Bernstein, a big data developer at Granite Telecommunications. Getting a big data pipeline architecture right is important, Schaub added, because data almost always needs some reconfiguration to become workable through other businesses processes, such as data science, basic analytics or baseline functionality of an application or program for which it was collected. The process of moving data from one data store to another becomes harder when there are multiple data stores involved and they’re spread across on-premises and public cloud data centers. This environment, Narayana said, is common these days as large enterprises continue migrating processes to the cloud.
The classic steps involved in a data pipeline are extract, transform and load (ETL). “After extracting the data,” Vilvovsky detailed, “it must go through a cleanup process where only the necessary data fields are left and converted into formats suitable for computation. Multiple data sources may be joined by combining and aggregating procedures.” Data is first generated by a user or process and requires movement to some type of database. These steps are known as collection and ingestion.
