p-values and the future of statistical inference

A Q&A with Ron Wasserstein of the American Statistical Association

@tandfSTEM recently ran a live Twitter Q&A with Ron Wasserstein, Executive Director of the American Statistical Association, on p-values and statistical inference. Ron was the lead author of the ASA’s p-values statement, which since its publication in June 2016, has generated much debate how research is conducted and what it will look like moving forward.

Let’s kick off with a bit of background about you and your career.

My career in statistics includes many years as a professor of statistics and an administrator at @WashburnUniv. The past 10 years I have been executive director of @AmstatNews. My involvement in the p-values discussion began in 2014 and is certainly a passion.

Can you please tell us the main things that prompted you to develop and publish the ASA’s p-Values statement?

The statement was needed because statistics was being blamed for statistical malpractice associated with p-values, because of rising concern about reproducibility in science, and because these issues have been known for a long time but change was not happening. The ASA statement on p-values is available to everyone online thanks to the @AmstatNews and @tandfSTEM publishing partnership.

What was the main impact you hoped the statement would have on how people conduct research in the 21st Century?

We hoped the ASA statement would jumpstart a conversation about changes needed in the culture of use of statistics in science. And it seems to have done so. The statement has been downloaded over 220,000 times and cited 750 times according to Google Scholar.

That’s certainly quite a reach. And what real-world impact have you seen in light of the statement?

I’ve been invited to speak on the topic by businesses, government agencies, and universities. Discussions are happening in places where statisticians or users of statistics are trained. There seems to be greater awareness of the issues.

I’ve heard some great anecdotes, including one about a major medical school in California that is changing its stats curriculum based on the p-values statement. But candidly I can’t tell if real change has happened yet. I’m eager to hear what others are seeing and hearing.

How do you feel the academic community has reacted to the p-values problem? Do you feel the discussion has been healthy?

My sense is that good conversations are taking place in academe about how to rethink teaching and research with regards to statistical inference. The issues to be addressed are not trivial, as anyone who teaches statistics or collaborates with researchers can attest.

There’s been a lot of press coverage about p-values and questions of significance in research, but how well do you think the issue has been communicated to the public at large?

Communicating about statistics in general and p-values in particular is difficult. Anything involving probability is immediately difficult to explain, and p-values are especially nonintuitive. Fortunately, there are some great writers like @ReginaNuzzo , @Monya_science , @juliaoftoronto, @cragcrest, @julierehmeyer , @StevenZiliak and others who are very good at explaining them.

The ASA recently held a follow-up symposium focusing not on the ‘don’ts’ of p-values, but on the ‘do’s – in other words, what should be done instead. What did you learn?

The discussion of alternatives is ongoing, and will be highlighted in special issue of The American Statistician, if you want to contribute to this discussion, see the call for papers at http://bit.ly/p-Values

Not surprisingly, there is no one solution to rule them all, and there never has been, even though p-values came to be that for some. Inference is hard. Data is often very noisy. We should bring all of our tools to bear.

Bayesian approaches, decision-theoretic techniques, & other tools are available. But first we have to break out of the culture of dichotomous thinking springing from the (mis)use of p-values, and teach people, as @StatModeling says, to embrace the uncertainty.

The nature of uncertainty and the ways we can measure it vary greatly across research domains. Nonetheless, uncertainty is an inherent feature of most research. Statistics helps control and quantify that uncertainty.

Do you have any reading recommendations on this topic? What have you seen that’s inspired you?

@StatModeling blog is a great source for information on p-values and inference (and so much more!). I love the @simplystats blog, and the Xian’s Og blog as well.

For a broad overview of science research issues, Richard Harris Rigor Mortis. Also, @NateSilver538 The Signal and the Noise, Ray Hubbard Corrupt Research, Herbert Weisberg Willful Ignorance, Ziliak and McCloskey The Cult of Statistical Significance.

What are your own top tips for effectively communicating uncertainty both within academia and to a wider audience?

I’ve been thinking lately about communication in terms of how to avoid publishing irreproducible research. There are questions researchers and readers of research should ask, especially when results are advertised as “ground-breaking.”

Here are some examples of good questions (and none of this is original with me.) What was the quality of the study design? What is the purpose of the research? Explore, confirm, find causes, make predictions, etc.?

Does the research report on the decisions made analyzing data that affect the statistical results, such as dealing with outliers or missing data? Does the research indicate how much the results differ if other decisions had been made?

What analyses were done but not included in the research report? How do the results of the research square with prior knowledge? Have ways of incorporating prior knowledge into the analysis been considered?

Is there a plausible mechanism to explain the outcome? (Even better if that mechanism is posited before the research was conducted.)

Many thanks to Ron for taking part in this Q&A for @tandfSTEM and to all those Twitter users who followed and took part in the conversation.

See also The American Statistician special issue call for papers, Statistical Inference in the 21st Century: A World Beyond P<0.05 (Deadline: 15 February 2018)