The Future Of Big Data

The last five years have seen tremendous advances in our ability to collect and analyze data, with many organizations entering the higher echelons of maturity. One new study by Progress Software shows an increase in big data adoption among organizations from 50% to 61%, and with new technologies being introduced seemingly every day, it will like soon become accepted that all facets of decision making are driven by data in every industry.

The next five years promise even more exciting developments in the data arena. We asked eleven experts in the field from organizations leading the way with their data initiatives what they think will be the major advances over the next few years.

Data is exploding, and the next 5 years will witness what one may call the renaissance of data science. New sources of data will generate an enormous amount of content, and unfortunately (or fortunately) there will be as much signal as noise in this data. With ‘Internet of Things‘ and ‘Artificial Intelligence‘, it will become a challenge for data scientists to isolate the ‘Right Signal’ at the ‘Right Time‘ (vs. today, where we still consider isolation of noise from the signal a challenge).

I am excited about the intersection of video/ consumption and analytics. Image and video recognition is still work in progress. There is a lot of exciting stuff that can be done with respect to what ML sees in videos and actual consumption/interaction response of the consumers.

It is all about using data to improve client experience. Developing the capability to interact with clients in a way that anticipates their needs takes some time to achieve. Robotics and artificial intelligence is a game changer and requires even more complex data transformations with natural language algorithms and machine learning capability. With the internet of things (IoT) getting immersed into daily life, there is lot more data produced by machines rather than humans and it requires the distributed power of big data platforms to capture huge data loads and make it available for analytics.

Deep Learning will have a major impact on how we think about learning patterns in big data in the far future. But that will first happen in a few companies and slowly trickle down to others.

In the near future, common statistical techniques will be more automated. For example, to see if two trends are related, the correlation coefficient can be computed and a fit for each trend computed automatically. Things like that will get easier with smarter tools.

I’m very excited to see more progress in the area of end-to-end differentiable network architectures. I feel that our rate of innovation will greatly accelerate if we could somehow also offload the learning of network architectures to machines.