Harnessing data for improved customer service and smart urban planning
The EDH was used to combine network topology (GIS) data with terabytes of DSL performance (time series) and electrical line test data to grade the quality of every line in the network. This helped indicate if slow speed was a network issue or a customer issue. Using this network analysis, the probability of a successful outcome of an engineer dispatch could be predicted, reducing wasted in-person engineer visits.
BT is one of the largest telecommunications providers in the world, with £24 billion (US$33.5 billion) in revenue in 2017.
The ability to broaden and deepen customer relationships is key to achieving sustainable, profitable growth in today’s competitive landscape. In order to meet customer expectations, it is essential for the company to know who their customers are, what services they are using and how those services are performing. Maintaining the quality and integrity of those data assets is a challenge.
Several years ago, Phillip Radley, chief data architect at BT , was having a discussion with colleagues about the next iteration of a critical extract, transform, and load (ETL) “pipeline.”
In the legacy environment, business client records were spread across multiple databases. They needed to be reconciled and updated daily with Dun & Bradstreet data in order to provide business units with the most relevant and up-to-date information.
Nearly one billion records were being compared and reconciled daily, and BT’s legacy ETL platform, built on a traditional relational database, couldn’t keep up with the pace. It was taking more than 24 hours to process 24 hours’ worth of data. Consequently, BT’s business units were working with day-old data at any given point in time.
The team initially had a proposal to re-platform the system to a new relational database.
“But as we sat down, our discussion turned to [Apache] Hadoop. We realized we basically had a data velocity problem. We had to process the data faster and increase the volume that we could ingest—both of which Hadoop excels at,” Mr Radley said.
BT engaged Cloudera to install a production-ready Hadoop cluster that replaced the batch ETL application with MapReduce[1] routines, and went from PowerPoint to production in nine months.
The company wanted its Linux administrators to manage the data platform instead of hiring new talent. Cloudera provided the required training saving the company time and money.
“The Cloudera University training course was not only high quality, but also the trainers were able to understand what we were trying to accomplish and helped ramp up the team quickly. The same people who run our 30,000 Linux servers also now run Hadoop, and they can do that on top of their other responsibilities,” said Mr Radley.
The new enterprise data hub (EDH) approach could not only solve BT’s immediate ETL problem, but it also helped tackle a host of big data challenges to help BT fast-track the delivery of new offerings.
BT has 1,900 operational systems and several of the world’s largest data warehouses. The EDH runs below the operational systems and the systems extend their data into the EDH. The data can then be shared and exposed as required. This unified, cost-effective infrastructure enables BT to gain unified views of its data across its multiple business units.
The platform also provides the ability to combine batch, streaming, and interactive analytics and allows business intelligence (BI) teams to perform SQL queries on the data.
Additionally, the environment enables the company to extend data retention from one year to more than 10 years when needed and implement innovative knowledge management use cases.
Security and stability were vital to the platform’s success. Security had to be as good as business-as-usual security.
