A Statistical Saga at a "Top" Journal
Even the “best” scientific journals can struggle with basic statistical literacy, to the detriment of science. We’ve known this for a long time (consider David Allison’s work trying to get journals to correct elementary statistical errors).
A recent case that came to my attention:
Richard Addante is a neuroscientist at Florida Tech. He and his colleagues did a study trying to replicate a prior finding on episodic memory (preprint here; published version here). The exact details of the study aren’t important — what’s important is what happened when they submitted their study to Nature Communications, specifically to an ongoing series on replication.
During the initial round of peer review, Reviewer 3 made the following comment (see the first doc here):
In other words, the reviewer was asking for a calculation as to statistical power: the ability of the sample in question to detect the effect that was seen.
The editor’s initial revise-and-resubmit email directly mentioned Reviewer 3’s comment (while confusing a “sensitivity analysis” with a post-hoc power analysis!):
The problem? Post-hoc power analysis isn’t useful or informative. You can plan a statistical power analysis in advance of a study, but once the study has already occurred, it is meaningless to ask, “Did this study have a sufficient sample size to uncover the effect observed here?” This is fairly uncontroversial (see Heinsberg and Weeks 2022, Hoenig and Heisey 2012, and Zhang et al. 2019).
Why? After a statistically significant effect has already been observed, it isn’t meaningful to ask whether the sample size in question would have had a particular chance of detecting that effect. We now know that it *DID* detect that effect.
***
The authors revised their paper to include a post-hoc power analysis, despite believing it to be meaningless.
But on a second round of peer review, a fourth reviewer pointed out (correctly!) that doing a post-hoc power analysis wasn’t useful or informative (see the third file here).
Even though the post-hoc power analysis had been done at the demand of the Nature Communications editor and Reviewer 3, the Nature Communications editor now used this review as a reason to reject the study for publication!
Needless to say, it was frustrating and arguably unethical for a researcher to be told:
do an inappropriate post-hoc power analysis thanks to Reviewer 3; but then,
we’re rejecting your paper in part because Reviewer 4 pointed out that post-hoc power analyses are wrong.
Moral: Don’t count on journal editors to have a correct understanding of basic statistics. And more broadly, the scientific journal system needs a better way to handle statistical issues like this.
There is a lot of blind leading the blind in peer review on statistical issues. It is a challenging issue as many quantitatively savvy scientists sometimes advocate bad statistical ideas but they might generally be good on other quantitative issues in their discipline.
Great discussion on the problems with post-hoc power a few years ago when surgeons insisted that post-hoc power analysis was needed and vehemently defended it.
https://discourse.datamethods.org/t/observed-power-and-other-power-issues/731/13
Something very similar happened to us. Although in our case, we were using an unusual statistical technique called "conformal prediction", so maybe we can cut the reviewer some slack. We explained the purpose of the technique very clearly, but the reviewer totally misunderstood the technique. After some protest, we convinced the journal to bring in a statistical reviewer. The statistical reviewer also didn't understand what we were doing, and thought we were "cheating" to get the result we wanted. One of our co-authors was Michael I. Jordan, one of the leading statisticians alive today. After more protest from faculty on our author list, the journal said they couldn't accept the publication due to the poor response from the statistical reviewer. Instead of fighting it further we decided to take the article elsewhere since it wasn't getting a fair treatment. Maybe I'll write up an account of the ordeal. The worst part was the journal made our first author spend many hours reformatting the article to the journal's standards, only to reject it because the statistical reviewer was clueless about what we were doing. So a lot of time was wasted. Very frustrating experience!