Building A Lakehouse Datastore Like Uber

In the IT world, solving one gnarly problem can lead to greater complexities down the road. Each incremental step can make things a bit better but can also just as easily expose discontinuities in scale between different parts of a platform.

Storage techies have had to wrestle with issue for the past decade as there has been a deluge of data that needs to be collected, stored, accessed, processed, and analyzed and a mix of structured,  unstructured, and semi-structured data.

When data first started getting big, companies used data warehouses for storing structured data, such as credit card, sales, and demographic information, which doesn’t change all that often and which sits in a standardized format with rows and columns and is easily accessible. With the rise of unstructured data – which accounts for almost 90 percent of all of today’s data and which does not fit easily and searchably into a spreadsheet and includes everything from images and videos to emails to telemetry data from IoT devices and other machine-generated information – enterprises constructed data lakes, which can store more data and do so at a lot lower cost than traditional block and file storage while also supporting complex queries against that data.

However, despite the benefits of data lakes, they come with their own issues and we have written quite a bit about how vendors like Dell EMC, Hewlett Packard Enterprise, Pure Storage, and Hitachi Vantara are trying help organizations get control of unstructured data. As the amount of data grows so does the complexity, particularly when AI and machine learning are thrown into the mix. There are concurrency, performance, data reliability, and data management issues that are still problematic.

Vinoth Chandar saw all this as a senior staff engineer and manager at fast-growing Uber from 2014 to 2019.

“We had a warehouse which had all the different advanced transaction capabilities and whatnot, but we couldn’t store all of our data in that,” Chandar tells The Next Platform. “We had a lake which can actually store a lot of the data and maybe even do a lot of high-scale data processing, but the data management of these features weren’t there.”

For example, Uber had a central database holding much of the ride-sharing company’s data regarding trips and, given the rapidly changing nature of the trips due to such issues as weather, Uber needed to be able to quickly replicate trips that are happening to its warehouse and leverage dashboards and queries on top of it.

“There are thousands of people running Uber who are operating in cities need that data to be able to make decisions on the ground,” he says. “We had volumes on the trips data that we could no longer fit in the warehouse that we were using, at least not in any cost-effective way. We needed to return to a lake, but the lake had zero ability to do updates. I couldn’t take trip changes in an upstream database and apply it to the lake directly. The lake can only store files and then I can write files and read files. There’s no intelligence to be able to absorb updates.”

At Uber, engineers were looking for the performance and speed of databases and the scale of data lakes to enable them to run their AI and machine learning workloads and reach near real-time capabilities for estimated arrival times for vehicles to food recommendations.