When Has a Data Analyst Succeeded?

I read a thought-provoking blog post by Roger Peng of Simply Stats (and professor in the Department of Biostatistics at the Johns Hopkins Bloomberg School of Public Health) entitled “What is a Successful Data Analysis?“. It’s an interesting question – data Analysis is concerned with measuring performance of people, processes and policies, but is there a widely-accepted measure of success of Analysis itself?

It seems like success in Data Analysis should be relatively straightforward to define, right? We all know good analysis when we see it, or at least we think we do. If you read Peng’s entire post, though, you’ll see that it’s tricky. If you’re tempted to use ‘veracity’ as a yard stick, how do you know whether the analysis is ‘true’ or not? If you feel like adherence to ‘best practices’ should be how we measure success, who defines what’s ‘best’, and would most people agree? In analysis there can be many ways to skin the cat, and multiple different findings can emerge from the same set of data.

Ultimately, he proposes the following definition for success for data analysts:

What do you think? Do you agree with this definition? As Peng’s audience, do you accept his selection of ‘acceptance’ as the sole criterion for success? If so, can you explain why this definition works for you? If not, do you think there’s another sole criterion, or do you think there are multiple criteria that should be used instead? Or maybe you think it’s not something you can define in general terms.

I want to like this definition, mostly because of the importance it places on communication and making an impact on other human minds, but unfortunately I just can’t accept it. It’s similar to leadership, in my view: is a ‘good’ leader someone who just gets people to follow him or her? If so, every tyrant and despot in human history would fit this definition.