Data Platforms
Like pipelines delivering fuel to generate energy for human advancement, data platforms have been used for decades to deliver data to optimize business processes. These technologies are loosely categorized under the terms – ESB (enterprise service bus), ETL (extract transform load), EDW (enterprise data warehouse), and BI (business intelligence). They have reincarnated multiple times over the last three decades to serve application development paradigms, data agility, and deployment choices. Let’s look at the past three eras to envision what lies ahead.
The pre-Big Data (the 90s to early-2000s) – The slow-moving sludge era
This was the era of waterfall application development and client-server architectures. Data was structured and usually rigid. Deployments were on-premise powered by expensive storage and compute. RDBMS was king of this era and SQL achieved mainstream adoption within the enterprise. A typical enterprise data management stack would look as follows –
In other words, this was the era of traditional software vendors, selling proprietary software to enterprise IT with SQL as the primary lingua-franca. IT was a cost-center, business leaders knew and could measure the ROI on data.
The open-source Big Data (the late-2000s to mid-2010s) – Adapting Yahoo, LinkedIn, Netflix technology for the Enterprise era
This era built upon the Agile movement driven by web-scale pioneers. Data was the new oil and fluidity mattered. Cloud deployments were in their infancy. A radically different data architecture emerged – based upon commodity hardware, cheap storage, and focussed on scale. These were driven by hyper-scale applications such as Google’s Search, and Amazon’s eCommerce. Engineers at application companies such as Yahoo, Facebook, and LinkedIn incubated and open-sourced their counterparts. Silicon Valley VCs jumped on the bandwagon as they got product market fit for free.
A typical enterprise data management stack would look as follows –
This era saw the rise of many NoSQL solutions under the open-source movement, but the crown belonged to Hadoop and the vendors that supported the ecosystem – Cloudera, HortonWorks, and MapR. Data sources proliferated across use cases such as Mobile, IoT, and SaaS. And vendors started touting AI/ML to the rescue to further attempt to mine value out of data.
