Innovating At The Speed Of Trust: Building The Privacy-Aware Data Pipeline

Data Pipeline initiatives allow companies to generate and act on insights from their investment in Big Data initiatives. By radically transforming the velocity at which applications and analytics can integrate new data, organizations can more easily react to new opportunities, make decisions, and transform key business processes.

BigID pioneered data discovery for personal data at rest — and we’re now extending those discovery and data intelligence capabilities to data in motion. The BigID platform now supports data pipelines so that companies can monitor sensitive data in motion on data streaming platforms including Kafka, Kafka Connect, AWS Kinesis, and FTP. Discover and classify data in motion, integrate consumer consent correlation across data streaming platforms, and extend privacy insight to high-speed pipelines.

Data streaming capabilities from tools like Apache Kafka, AWS Kinesis, and Confluent’s Kafka Connect enable organizations to utilize the data from existing mobile applications, IoT devices, business applications, data lakes, and other event sources to rapidly gain insight and trigger responsive business processes.

Global privacy regulations such as CCPA and GDPR, meanwhile, have introduced specific requirements for protecting new categories of personal information — while limiting how it can be used, shared or processed. To meet these regulations and approach data processing ethically, it’s critical that organizations integrate privacy insights into data pipeline initiatives.

Fines may have a material impact, but customers and consumers will move elsewhere if they lose trust in a company’s brand because of privacy violations.

Given the velocity, volume, and complexity of data flowing through today’s data pipelines, it’s a huge challenge for companies to ensure that they are doing the right thing with data — both in respect to data privacy, and by operating with privacy by design principles when developers and analytics teams build applications. In order to do so, organizations need to be able to:

For one thing, losing track of personal data as it streams across the enterprise — or worse, using it in ways that violate consumer privacy protection — can undermine trust in ways that far exceed the cost of regulatory fines and statutory civil liabilities.