Over the past decade, big data analysis and applications have revolutionized practices in business and science. They enabled new businesses (e.g., Facebook, Netflix), to disrupt existing industries (e.g., Airbnb, Uber), and accelerated scientific discovery (genomics, astronomy, biology).
Today, we are seeing glimpses of the next revolution in data and computation, driven by three trends.
First, there is a rapidly growing segment of the economy (e.g., Apple, Facebook, GE) that collects vast amounts of consumer and industrial information and uses this information to provide new services. This trend is spreading widely via the increasing ubiquity of networked sensors in devices like cell phones, thermostats and cars.
Second, recent advancements in deep neural networks, reinforcement learning, and big data machine learning systems have unlocked remarkable AI capabilities ranging from visual perception to superhuman game playing capabilities to saving on power consumption in datacenters, and learning complex locomotion tasks.
Third, a growing number of devices such as security systems, drones, and self-driving cars are autonomously or semi-autonomously taking action in the physical world.
These trends point to a future in which
computing infrastructure senses the world around us, ingests information, analyzes it, and makes intelligent decisions in real time on live data. These abilities can fundamentally improve how both humans and machines interact with the world and each other, while raising critical new issues in security and privacy.
These are the challenges we set to address at RISELab, a new five year lab we started at UC Berkeley, and which follows AMPLab, the home of many successful open source projects, such as Apache Spark, Apache Mesos, and Alluxio.
There are currently few mature and widely-used examples of real-time decision-making on live data. Two examples stand out, as they helped create enormously successful industries: high-frequency trading (HFT) and real-time targeted advertising.
HFT is now a key component of today’s financial markets, responsible for billion-dollar trading decisions daily. While there is relatively little public information about the performance of these systems, clearly there has been a continuous drive towards real-time ad targeting and bidding with sub-second latencies. These examples, custom-built for their virtual environments, hint at an even higher-impact future reaching into the physical world.
The combination of real-time intelligent decisions on live data with sensing and actuation will enable new categories of applications, such as real-time defense against Internet attacks, coordinating fleets of airborne vehicles, robot assistants for the home, and many others. These applications are all data-hungry and require real-time, intelligent, secure decision-making systems, with techniques for sharing data that preserve confidentiality and privacy, and provide robustness to attacks and security breaches.
Next, we discuss the desirable attributes of a decisions system to enable the above applications in more detail:
Intelligent: Decisions that take place in uncertain environments and are capable of adapting to context and feedback are inherently non-trivial. Examples of such decisions are detecting attacks in the Internet, coordinating a fleet of flying vehicles, or protecting the home. One promising approach to implement intelligent decisions is Reinforcement Learning, which has recently been used with great success in varied applications from beating the Go worldchampion to robotics.
Real-time: Real-time refers not only to how fast are the decisions rendered, but also to how fast are the changes in the environment incorporated in the decision process. For instance, in the case of intrusion detection, we would like to create an accurate model of the attack in seconds, and then decide which are the offending streams and and drop their packets.
