Beyond Big Data: Digital Transformation Comes to Financial Services

If this year’s LendIt USA conference in New York City is any indication, we’re at the beginning of a new era in Financial Services. We’re moving beyond Big Data into a new era of Digital Transformation in which Machine Learning radically improve data analysis, enabling new types of products, services, and businesses.

Allow me to explain.

The era of Big Data hit the mainstream media around 2005. That’s the year when Yahoo! built its first iteration of the Hadoop data store and Roger Mougalas from O’Reilly Media used the term “Big Data” to refer to data sets so large they were unmanageable with traditional tools. Big Data had its roots, though, in the late 1990’s with OLAP data stores.

Banks have long recognized that they have access to more data on their customers than most other businesses. They collect data from online banking, Web advertising, branch visits, call centers and other sources, always mindful of the opportunity to learn more about customers and market trends.

In 1998, at the large regional bank where I worked, we struggled to recognize a life event represented by a customer’s unusually large deposit in time to do anything about it. Banks were sitting on a digital gold mine. They knew it, but they weren’t sure yet how to leverage it into actionable information.

In the Big Data era, organizations began heavily investing in new IT solutions (such as data warehouses, integration layers, and Hadoop) for consolidating, storing and analyzing this vast quantity of data.  When they analyzed data from across the silos they had built, they began to get a clearer, more holistic view of their customer’s behaviors and needs for the first time. From the board level down, Big Data was and still is recognized as a strategic initiative worthy of investment and attention.

But data analytics is about to get a new technological boost that makes the Big Data analysis of a few years ago seem like a local commuter rail compared to a bullet train.

What’s new now is the use of Machine Learning to accelerate analysis, to make analysis progressively smarter and more effective, and to automate analysis that years ago could only have been achieved by a roomful of hardworking, highly paid analysts or data scientists poring over data and delivering results in months or weeks instead of hours or seconds.

Startling advances in Machine Learning are showing up all kinds of business and applied analysis, ranging from genomics to sports betting to finding your best customers, the Keepers.

We’re now able to find signals in what previously would have been only a vast sea of noise.