Using data to drive a smarter way to faster insights

Growing expectation around getting more business value from data in a cost-effective manner is driving change across industries. As organizations collect data of increasing volume and variety, teams of data management analysts and data scientists are also formed to put this data to good use.

A typical data science project begins with extracting profile data (customer, product, organizations, stores, suppliers) from some master data management system and loading it into an S3 or some other data warehouse. The next task usually involves spinning up and managing servers to run analytics applications. After that, the data science team will develop an application to execute the query, and use various mechanisms to bring results and updates back to the master profiles. This process is often repeated if the data changes.

The problem with this approach lies with the significant budget and resources required. Data cleanup and data movement are complex endeavors. It’s difficult to integrate and synchronize master data management and analytic systems in both directions. Moreover, the accuracy of insights generated are debatable without a clean and reliable data foundation.

Let’s take the example of a healthcare organization that’s looking for better ways to engage with their members and offer them relevant wellness programs. Their goal is to focus on their patients’ overall well-being and quality of life. Healthcare organizational objectives include lowering readmission rates, ensuring adherence and delivering quality care. Delivering such care and targeting the right member with the right programs requires deeper understanding of their members.

Members of healthcare organizations are demanding consumer-like, personalized experiences, as well as forcing healthcare providers to shift towards a patient-centric approach. This approach focuses on building long-term relationships in order to understand patients’ needs, behaviors and preferences.

Now let’s look at retail. Many retailers are taking advantage of algorithms, like “collaborative filtering” to provide purchase recommendations for their customers. This method makes automatic predictions about a customer’s interests through the collection of preferences from many users with similar profiles. This allows organizations to create offers that are more relevant, compared with generic, demographic-based segmentation. Healthcare organizations task data science teams to design similar prototypes to send personalized promotions to their members.

Data scientists often waste much of their time in data cleanup and ETL activities, which is not the best use of their talent. Moreover, the time and resources required to setup hardware, applications and prototyping are quite prohibitive. It’s not uncommon for data science teams to spend three to four months developing any working model or application.