Data-driven management has risen sharply from a decade ago, when Thomas Davenport wrote Competing on Analytics.Data is now the critical tool for managing many corporate functions, including marketing, pricing, supply chain, operations, and more. This movement is being further fueled by the promise of AI and machine learning, and by the ease of collecting and storing data about every facet of our daily lives. But has the pendulum swung too far? Are managers relying excessively on data to guide their decisions, abdicating their own knowledge and experience?
One possible solution may be found in Agent-Based Simulation(ABS), a novel approach to solving complex business problems through computer simulations. One of the most appealing aspects of ABS is that it combines domain expertise and data. The domain expertise is used to define the structure of the simulation, which is unique to each business problem. The data is used partly to refine the details of the simulation, and partly to ensure that as the simulation runs, the resulting outcomes match real-world results. With this approach, the manager’s expertise regains the primary role, and the results of the simulation can be analyzed by the manager and data scientist together.
Data is now the critical tool for managing many corporate functions, including marketing, pricing, supply chain, operations, and more. This movement is being further fueled by the promise of artificial intelligence and machine learning, and by the ease of collecting and storing data about every facet of our daily lives.
But is the pendulum starting to swing too far? As a practitioner and teacher of predictive analytics, my greatest concern is what I call the “big data, little brain” phenomenon: managers who rely excessively on data to guide their decisions, abdicating their knowledge and experience.
In a typical big data project, a manager engages an internal or external team to collect and process data, hoping to extract insights related to a particular business problem. The big data team has the expertise needed to wrangle raw data into usable form and to select algorithms that can identify statistically significant patterns. The results are then presented to the manager through charts, visualizations, and other types of reports. This scenario is problematic because most managers are not experts in data science, and most data scientists are not business experts. Addressing this dichotomy requires individuals who can “serve as liaisons” between the two, as Todd Clark and Dan Wiesenfeld suggested in a recent HBR article.
This, however, is simply a palliative that does not resolve the underlying problem. As Tom Davenport wrote in HBR in 2006, the year before publishing his seminal book, Competing on Analytics, “For analytics-minded leaders, then, the challenge boils down to knowing when to run with the numbers and when to run with their guts.” Rather than reducing reliance on intuition, the advanced methodologies of big data require managers to use even more intuition to make sense of the growing number of outputs and recommendations being generated by data models.
