Hi, thank you for an interesting post! Visibility of replication studies is definitely an important topic to bring into wider awareness. We are currently working on a project at FORRT, focused on this exact problem: https://forrt.org/marco/. If you are open to it we would welcome feedback and/or to share notes/brainstorm!
I think it’s wrong to view replication as having a binary outcome — confirmed or disconfirmed. For one thing, that’s not how statistical significance works. There are two potential aspects to replication — increasing the data, and validating the methods. These operate somewhat independently. It’s possible that a replication study produces results that are statistically different from the original study, even though it “confirms” the original findings — that is still an important observation, because it demonstrates that there are significant factors that are not adequately controlled. It’s also possible that a replication study produces results that are not statistically different from the original study (i.e., the hypothesis that the two studies had the same outcome distribution is not disconfirmed) but the results still are less statistically significant than the original study. And it’s also possible that the combined results of the original and the replication study produce more significance or more analyzable data than either alone.
For all of these reasons and others, I also agree with Stuart that it’s wrong to look at the value of replication studies as simply “probability of failure to replicate” times “impact of failure to replicate”. If a study is very influential, there could be significant value in *confirming* it, for example. And just expanding the amount of data and therefore the confidence intervals for outcome variables could also be worth a lot.
Very interesting discussion and I hope this leads to fruitful allocation of the promised billions.
The model is obviously sensitive to the replication rate as you can see from the dashboard on the IFP piece (https://ifp.org/how-much-should-we-spend-on-replication/). The fact that you've included a way to computationally reproduce the piece instantly is great and I can't find too much fault for that reason. I hope readers put in their own estimates. (As a side note, journals attempted making automatic dashboards and it had very little uptake by authors from what I understand. So this is truly great and should be imitated.)
Now to the critique. You say, "Counting all fully overturned cases and half of the weakened cases yields an estimated (and potentially conservative) ~11% rate of genuine unreliability across the literature." So you're using 1 * O + 1/2 * W = 11% where O is the rate of papers overturned and W is the rate of weakened papers from Brodeur et al., 2024. This is the rate of what you're calling unreliability.
1. Unreliability is not a statistical term.
2. Brodeur et al. is a great paper but it is not about collecting new data, which is the traditional definition of replication and where the 50% you cited earlier comes from. Brodeur has made an excellent point that computational and robustness replication is cheaper and should be attempted first, but his paper is not a measure of replication and not a counterpoint to the 50%. You could easily have a computationally reproducible paper that has no tether to reality. It means the code runs and gives the author's result. Robustness means the results hold up to other assumptions. It is not collecting new data and does not test intersubjective agreement.
3. Using the ordinary, well supported and in many ways conservative estimate of 50%, the ROI jumps from 0.75x to 3.4x. The chart suggests ROIs from 1.0 to 2.2x , corresponding to an assumed replication rate between 0.15 and 0.33 and suggesting between 0.9 to 2.1% budget at NIH. In other words, you don't think a 50% replication rate is worth discussing even though the dashboard says, "Probability original study isn't replicable," not "Probability original study is unreliable." We have a rate of original studies being replicable and it's about 50%. There are of course caveats and nuance but a new term "unreliability" risks confusion.
As it concerns this piece, I think you need to be clear about what your assumptions are and whether they are meant as a floor on the problem, or a good estimate of it because it affects everything thereafter.
Thanks for your comments, Alex. Glad you found the dashboard useful, it was fun to build (and I am still mourning the loss of distill.pub, which was a wonderful attempt to incorporate more interesting/dynamic visualization into academic computer science research).
To your critiques:
1. True, unreliability is not precise, but it's an attempt to capture what we actually care about, which is: "is this paper a solid foundation to build upon?" Published replication rates often, though not always, capture something much broader (e.g. "does a follow-up paper obtain p<.05") that I wanted to draw a distinction with.
2. I agree that {computational reproducibility + robustness to alternative assumptions} is a lower bar than replication on new data (though in a computational domain – where p-hacking likely occurs at the level of model specification rather than data collection – the presence of robustness checks is closer to "new data replication" than I think you give it credit for). But even if you don't want to concede that point, it's worth highlighting that Brodeur's paper also finds non-replication+robustness rates closer to 30-50%. The 11% comes from additional work that an expert colleague (not Brodeur) did, in which he went paper by paper within Brodeur's sample and subjectively assessed whether the findings of the replication substantially weaken or overturn the original findings. We found that even with a high top-line replication rate, a significantly lower number weakened the results enough that we would caution people against building on the results. Even so, it is definitely possible that number is conservative, e.g. if failure to replicate in lab sciences more often reflects comprehensive incorrectness than it does in Brodeur's sample.
3. Fair point. The dashboard probably should say "is unreliable" rather than "isn't replicable". The fact that I conflated the two here is good evidence that the new term risks confusion! I will say that: (1) it's true that I don't really think the 50% rate is a great estimate for the proportion of papers that are fully wrong; and, perhaps more importantly, (2) if you use 50%, it's probably also worth dropping the "expected citations" parameter back down to the ~median of 50, which returns the ROI to ~0.7x. There's pretty good evidence (e.g. Bik, Casadevall, & Fang) that higher quality journals – and thus more influential papers – have lower rates of fraud/misconduct, so a funding program targeting high-impact replication opportunities should probably not assume that base rates apply.
But overall, I wanted the report to be a framework for thinking about the problem not just a fixed answer, so it's great if people to use the dashboard to form their own (somewhat grounded) intuitions that differ from mine.
This could go on, I think. Quickly, I think it's safe to say that we've got one foot in statistics and one in opinion, and my opinion about what we should be using replication for is different from yours. More specifically since your post is about ROI, my opinion is that we need an estimate that maximizes ROI and your estimate is more of a lower bounds on how bad the problem could be.
Scientifically, I don't think you need any reason at all to replicate. It doesn't even need to be tied to some outcome. It might be better if it's not. The conflict with outcome -- someone's career usually -- is how we got into this mess. Having no predicted outcome is not realistic with public funds so ROI is a perfectly reasonable metric.
You're right that the weakest part of my argument is on robustness, which is hopefully empirical and has something to do with reality.
The only additional point I'll make, which also relates to what you're saying about 50% implying other changes to the parameters, is that giving us some ways to explore the dashboard via our priors doesn't mean the model isn't still opinionated. The dashboard has a built in functional form and shape parameter that is fixed. For instance, if I think non-replication, or unreliability for that matter, is best estimated by a normal distribution around 40%, the dashboard doesn't give that option. It gives what I would call roughly the minimum possible distribution for that mean. I don't have the equations to confirm but I think that's right. We at least know beta distributions are not unique to the mean, and beta is not the only reasonable choice.
All that's fine if it's clear how much opinion was used. I think that's my overarching comment that it's not clear enough what is certainly implied by the data. If you're willing to do a write up for a technical audience I would be enthused!
The suggestion seems entirely backwards to me. The studies we should replicate are those that have a lot of impact, as measured by accumulating a lot of citations. It would be inefficient to replicate studies early on, when we don’t know if they will have any impact. Could spend a lot on replication of stuff that doesn’t matter.
We could also require that journals add “forward citations” to old articles, linking to replication studies. That would be a way to increase visibility.
I think we all agree, David - the approach Stuart and I discuss here is not randomly replicating all brand new studies, it's identifying relatively new studies that show rapid accumulation of influence (and citations) and replicating them quickly before that influence becomes too baked-in.
And we do have an exchange in this piece about adding forward citations to replications, which both Stuart and I think is a very good idea
Hi, thank you for an interesting post! Visibility of replication studies is definitely an important topic to bring into wider awareness. We are currently working on a project at FORRT, focused on this exact problem: https://forrt.org/marco/. If you are open to it we would welcome feedback and/or to share notes/brainstorm!
A few more comments:
I think it’s wrong to view replication as having a binary outcome — confirmed or disconfirmed. For one thing, that’s not how statistical significance works. There are two potential aspects to replication — increasing the data, and validating the methods. These operate somewhat independently. It’s possible that a replication study produces results that are statistically different from the original study, even though it “confirms” the original findings — that is still an important observation, because it demonstrates that there are significant factors that are not adequately controlled. It’s also possible that a replication study produces results that are not statistically different from the original study (i.e., the hypothesis that the two studies had the same outcome distribution is not disconfirmed) but the results still are less statistically significant than the original study. And it’s also possible that the combined results of the original and the replication study produce more significance or more analyzable data than either alone.
For all of these reasons and others, I also agree with Stuart that it’s wrong to look at the value of replication studies as simply “probability of failure to replicate” times “impact of failure to replicate”. If a study is very influential, there could be significant value in *confirming* it, for example. And just expanding the amount of data and therefore the confidence intervals for outcome variables could also be worth a lot.
Very interesting discussion and I hope this leads to fruitful allocation of the promised billions.
The model is obviously sensitive to the replication rate as you can see from the dashboard on the IFP piece (https://ifp.org/how-much-should-we-spend-on-replication/). The fact that you've included a way to computationally reproduce the piece instantly is great and I can't find too much fault for that reason. I hope readers put in their own estimates. (As a side note, journals attempted making automatic dashboards and it had very little uptake by authors from what I understand. So this is truly great and should be imitated.)
Now to the critique. You say, "Counting all fully overturned cases and half of the weakened cases yields an estimated (and potentially conservative) ~11% rate of genuine unreliability across the literature." So you're using 1 * O + 1/2 * W = 11% where O is the rate of papers overturned and W is the rate of weakened papers from Brodeur et al., 2024. This is the rate of what you're calling unreliability.
1. Unreliability is not a statistical term.
2. Brodeur et al. is a great paper but it is not about collecting new data, which is the traditional definition of replication and where the 50% you cited earlier comes from. Brodeur has made an excellent point that computational and robustness replication is cheaper and should be attempted first, but his paper is not a measure of replication and not a counterpoint to the 50%. You could easily have a computationally reproducible paper that has no tether to reality. It means the code runs and gives the author's result. Robustness means the results hold up to other assumptions. It is not collecting new data and does not test intersubjective agreement.
3. Using the ordinary, well supported and in many ways conservative estimate of 50%, the ROI jumps from 0.75x to 3.4x. The chart suggests ROIs from 1.0 to 2.2x , corresponding to an assumed replication rate between 0.15 and 0.33 and suggesting between 0.9 to 2.1% budget at NIH. In other words, you don't think a 50% replication rate is worth discussing even though the dashboard says, "Probability original study isn't replicable," not "Probability original study is unreliable." We have a rate of original studies being replicable and it's about 50%. There are of course caveats and nuance but a new term "unreliability" risks confusion.
As it concerns this piece, I think you need to be clear about what your assumptions are and whether they are meant as a floor on the problem, or a good estimate of it because it affects everything thereafter.
Thanks for your comments, Alex. Glad you found the dashboard useful, it was fun to build (and I am still mourning the loss of distill.pub, which was a wonderful attempt to incorporate more interesting/dynamic visualization into academic computer science research).
To your critiques:
1. True, unreliability is not precise, but it's an attempt to capture what we actually care about, which is: "is this paper a solid foundation to build upon?" Published replication rates often, though not always, capture something much broader (e.g. "does a follow-up paper obtain p<.05") that I wanted to draw a distinction with.
2. I agree that {computational reproducibility + robustness to alternative assumptions} is a lower bar than replication on new data (though in a computational domain – where p-hacking likely occurs at the level of model specification rather than data collection – the presence of robustness checks is closer to "new data replication" than I think you give it credit for). But even if you don't want to concede that point, it's worth highlighting that Brodeur's paper also finds non-replication+robustness rates closer to 30-50%. The 11% comes from additional work that an expert colleague (not Brodeur) did, in which he went paper by paper within Brodeur's sample and subjectively assessed whether the findings of the replication substantially weaken or overturn the original findings. We found that even with a high top-line replication rate, a significantly lower number weakened the results enough that we would caution people against building on the results. Even so, it is definitely possible that number is conservative, e.g. if failure to replicate in lab sciences more often reflects comprehensive incorrectness than it does in Brodeur's sample.
3. Fair point. The dashboard probably should say "is unreliable" rather than "isn't replicable". The fact that I conflated the two here is good evidence that the new term risks confusion! I will say that: (1) it's true that I don't really think the 50% rate is a great estimate for the proportion of papers that are fully wrong; and, perhaps more importantly, (2) if you use 50%, it's probably also worth dropping the "expected citations" parameter back down to the ~median of 50, which returns the ROI to ~0.7x. There's pretty good evidence (e.g. Bik, Casadevall, & Fang) that higher quality journals – and thus more influential papers – have lower rates of fraud/misconduct, so a funding program targeting high-impact replication opportunities should probably not assume that base rates apply.
But overall, I wanted the report to be a framework for thinking about the problem not just a fixed answer, so it's great if people to use the dashboard to form their own (somewhat grounded) intuitions that differ from mine.
Wonderful! Thank you for the productive response.
This could go on, I think. Quickly, I think it's safe to say that we've got one foot in statistics and one in opinion, and my opinion about what we should be using replication for is different from yours. More specifically since your post is about ROI, my opinion is that we need an estimate that maximizes ROI and your estimate is more of a lower bounds on how bad the problem could be.
Scientifically, I don't think you need any reason at all to replicate. It doesn't even need to be tied to some outcome. It might be better if it's not. The conflict with outcome -- someone's career usually -- is how we got into this mess. Having no predicted outcome is not realistic with public funds so ROI is a perfectly reasonable metric.
You're right that the weakest part of my argument is on robustness, which is hopefully empirical and has something to do with reality.
The only additional point I'll make, which also relates to what you're saying about 50% implying other changes to the parameters, is that giving us some ways to explore the dashboard via our priors doesn't mean the model isn't still opinionated. The dashboard has a built in functional form and shape parameter that is fixed. For instance, if I think non-replication, or unreliability for that matter, is best estimated by a normal distribution around 40%, the dashboard doesn't give that option. It gives what I would call roughly the minimum possible distribution for that mean. I don't have the equations to confirm but I think that's right. We at least know beta distributions are not unique to the mean, and beta is not the only reasonable choice.
All that's fine if it's clear how much opinion was used. I think that's my overarching comment that it's not clear enough what is certainly implied by the data. If you're willing to do a write up for a technical audience I would be enthused!
The suggestion seems entirely backwards to me. The studies we should replicate are those that have a lot of impact, as measured by accumulating a lot of citations. It would be inefficient to replicate studies early on, when we don’t know if they will have any impact. Could spend a lot on replication of stuff that doesn’t matter.
We could also require that journals add “forward citations” to old articles, linking to replication studies. That would be a way to increase visibility.
I think we all agree, David - the approach Stuart and I discuss here is not randomly replicating all brand new studies, it's identifying relatively new studies that show rapid accumulation of influence (and citations) and replicating them quickly before that influence becomes too baked-in.
And we do have an exchange in this piece about adding forward citations to replications, which both Stuart and I think is a very good idea