Data Lake 3.0: What’s a self-driving car got to do with Data Lake 3.0?

This blog has contributions from: Vinod Vavilapalli, Wangda Tan, Gour  Saha, Priyanka Nagwekar, Sunil Govindan

You have probably wondered what makes a self-driving car intelligent to process the live camera feeds, navigate the busy streets and distinguish objects on the streets, such as cars, trucks, traffic lights or pedestrians? A self-driving car is a perfect example of a modern data application that combines big data with smart algorithms. To understand the underpinnings of such a modern data app, we will start with a recap of our blog series titled “Data Lake 3.0” (pt1,pt2,pt3,pt4,pt5,pt6) and then, conclude with the key takeaways from thekeynote demoin Data Works Summit, San Jose, 2017.

We are seeing the emergence of modern data applications, that exploit the big data; are architected to be micro-service based and containerized; are compute/GPU intensive, and deployed on a commodity infrastructure. Our Data Lake 3.0 architecture is at the cross-section of all these major trends and we want to walk you through a simplified example of a self-driving car. If you want to familiarize with what a Data Lake 3.0 is, you might want to refer topt1of our blog series.

A self-driving car generates massive amount of videos that need to be captured and stored in a centralized active archive for access by data scientists and analysts. This requires a storage layer that can scale to billions of files, exabytes and accessible by end users, while being Total Cost of Ownership friendly. Hadoop storage layer (Apache HDFS) powered by Hadoop 3.0 provides the erasure coding to store the data at half the cost (vs. 3 replica approach), while allowing linear scale and unified name space with NameNode Federation and View FileSystem. It hasdevice behavior analyticsbuilt in so that a slow commodity server and a slow commodity network switch will not interrupt a latency sensitive operation.

Now, a data scientist needs to train distributed deep learning models (by using frameworks like TensorFlow) that will process natural signals such as videos before the model gets deployed in the car and this is an ongoing task- the more it trains, the better the self-driving car gets. Training is a very compute intensive process.