2015: A Transformative Year for Big Data

Throughout 2015, big data continued to be big news. Across every industry and government sector, data analytics involving ever-increasing amounts of structured and unstructured data is changing the world.

At the start of the year, pundits were predicting one of two things. Either 2015 would be the year that the “big data fad” finally fizzled, or it would be the year that big data went truly mainstream. Having spent the year working with companies of all shapes and sizes, and speaking to probably hundreds of people involved in analytics projects, I have seen nothing to convince anyone that the buzz is fizzling out, but plenty that shows mainstream applications. Wherever you look, big data and analytics is taking place – across healthcare, crime fighting, finance, insurance, travel, transport, science, and entertainment. 2015 was the year big data broke out of the IT lab and became something that everyone wanted to be part of.

So here’s a roundup of some of the milestones and landmarks of 2015 for big data, the highs and – to keep things balanced – the lows as well.

The ever-increasing size

The most obvious is of course the ever-increasing size. Data continued to grow at a phenomenal rate, with just one example being the 1 trillion pictures being taken with digital cameras. Every day, 315 million of these are uploaded to Facebook. In fact, the number of users logging into Facebook in one day exceeded one billion for the first time in August – that’s one seventh of the world’s population logging into one network in a single day!

And that’s just a small amount of the data which is being generated by people. In fact, far larger volumes are being generated automatically by machines. Increasingly, due to the Internet of Things machinery, tools, and vehicles are capable of talking to each other, logging and recording everything that their sensors can measure. This year, driverless cars from many of the major automobile manufacturers took to the roads for trials. And giants of industry such as GE and Rolls Royce pressed on with developing and refining the “industrial Internet.”

2015: The year Big Data went mobile

2015 was also the year big data went mobile in a big way. We entered the year with, for the first time ever, more people using mobile devices than fixed line broadband to connect to the Internet. This seismic shift in the way we consume data has led to a widespread “repurposing” of the Internet, toward serving up data on the go. More than 14 billion cell phones were shipped this year. On top of that, mainstream consumers began to take wearable technology seriously by the for the first time, with Apple selling more than 3 million watches, and Fitbit shipping more than 4 million wearable fitness trackers.

In homes, too, the Internet of Things continued to grow in popularity. Devices like Nest’s smart thermostat have become an increasingly common sight, often thanks to deals with power companies  that see the devices fitted for free when customers sign up to a contract. Devices like these have shown that the efficiency they bring is a benefit to all parties.

Machine learning was hot

Machine learning was another hot topic in 2015. The rise in “cognitive computing,” which has been encapsulated as the development of computers that can learn for themselves rather than having to be programmed, has been one of the most frequent subjects of discussion. This year, IBM continued development and refining of its Watson Analytics engine, which puts artificial intelligence-driven big data analytics in the hands of businesses and individuals around the world.

Legislators catched up on the Big Data revolution

2015 also may be remembered as the year that legislators began to catch up with the big data revolution. In one court case that is likely to have far reaching implications, courts in Europe sided with Austrian citizen Mark Schrems, who complained that American businesses that were transmitting his data from the EU to the United States were not taking adequate care that his information would be protected from unauthorized use. This brought to an end the so-called “safe harbor” agreement, which previously codified that it was taken for granted that giant U.S. data-driven companies such as Google and Facebook could be trusted to look after our personal data. (In years to come, I predict we will look back on that sort of thinking the way we now look at people who used to think the Earth was flat!) Adapting to the hurdles that this ruling puts in their way undoubtedly will be a priority for data-driven businesses, moving into 2016.

Big Data's dark side

The dark side of big data was also brought into focus by the ongoing trend of large-scale data breaches and thefts. 2014 saw a greater than 50 percent increase in the number of people falling victim to these new hazards of the big data age, and although the figures aren’t in yet, it looks like 2015 will smash all records once again. Customers of U.S. pharmacy chain CVS, UK phone retailer Carphone Warehouse, dating site Ashley Madison, crowdfunding service Patreon, password management service Lastpass, and credit reference agency Experian, all experienced the fun and excitement of being informed their highly personal and often financially sensitive data had fallen into the hands of persons unknown, affecting many millions of customers.

Big data is suspicious

Not long ago, some people were quite open about their suspicion of the term “big data.” There were those who considered it a buzzword or fad that would soon die out, and said that those hawking it would move onto whatever became fashionable next.

Meanwhile another group of detractors, these ones harbouring no doubts over the ability of big data to change the world, warned it was more likely to become a tool of surveillance and oppression rather than positive economic and social change.

This year has certainly proven the first group to be mistaken. Investment in big data and analytical technology continues to grow at an ever-increasing rate. And the results are apparent in the flood of companies coming forward to talk about how their analytics projects are driving success and growth.

And the second group? Well, their fears are not as easy to dismiss. Nobody knows what tomorrow will bring. What I do know is that, of the hundreds of people I have spoken with about big data this year, the vast majority are confident that its potential for good outweighs its potential for bad. But that doesn’t mean that we shouldn’t continue to be vigilant. Governments as well as corporations must always be held to account to ensure they have our best interests at heart when we let them use our data.

With that in mind, 2016 is likely to be another boundary-shattering year for those of us lucky enough to be part of the big data revolution, and I, for one, am very excited about what it will bring.