51 Big Data Terms You Need to Know
With billions of bytes of data being collected daily, it’s more important than ever to understand the intricacies of big data. In an effort to help bring clarity to this field, we created a compiled list from our recent big data guides of what we feel are the most important related terms and definitions you need to know. (By the way, if you’re interested in this, you might also be interested in our AI glossary!)
Any terms you think we should add? Let us know in the comments!
Algorithm: A set of rules given to an AI, neural network, or other machines to help it learn on its own; classification, clustering, recommendation, and regression are four of the most popular types.
Apache Flink: An open-source streaming data processing framework. It is written in Java and Scala and is used as a distributed streaming dataflow engine.
Apache Hadoop: An open-source tool to process and store large distributed data sets across machines by using MapReduce.
Apache Kafka: A distributed streaming platform that improves upon traditional message brokers through improved throughput, built-in partitioning, replication, latency, and reliability.
Apache NiFi: An open-source Java server that enables the automation of data flows between systems in an extensible, pluggable, open manner. NiFi was open-sourced by the NSA.
Apache Spark: An open-source big data processing engine that runs on top of Apache Hadoop, Mesos, or the cloud.
Artificial intelligence: A machine’s ability to make decisions and perform tasks that simulate human intelligence and behavior.
Big data: A common term for large amounts of data. To be qualified as big data, data must be coming into the system at a high velocity, with large variation, or at high volumes.
Blob storage: An Azure service that stores unstructured data in the cloud as a blob or an object.
Business intelligence: The process of visualizing and analyzing business data for the purpose of making actionable and informed decisions.
Cluster: A subset of data that share particular characteristics. Can also refer to several machines that work together to solve a single problem.
COAP: Constrained Application Protocol is an Internet Application protocol for limited resource devices that can be translated to HTTP if needed.
Data engineering: The collection, storage, and processing of data so that it can be queried by a data scientist.
Data flow management: The specialized process of ingesting raw device data, while managing the flow of thousands of producers and consumers. Then performing basic data enrichment, analysis in stream, aggregation, splitting, schema translation, format conversion, and other initial steps to prepare the data for further business processing.
Data governance: The process of managing the availability, usability, integrity, and security of data within a data lake.
Data integration: The process of combining data from different sources and providing a unified view for the user.
Data lake: A storage repository that holds raw data in its native format.
Data mining: A practice to generate new information through the process of examining and analyzing large databases.
Data operationalization: The process of strictly defining variables into measurable factors.
Data preparation: The process of collecting, cleaning, and consolidating data into one file or data table, primarily for use in analysis.
Data processing: The process of retrieving, transforming, analyzing, or classifying information by a machine.
