One version of this that rarely gets discussed: null impact.
Funders increasingly ask for societal impact, and projects report it. But the reporting usually has no baseline, so it can't come out null. A paper becomes increased knowledge, knowledge becomes deeper understanding, understanding becomes public discussion, and discussion gets written up as impact. Nothing can fail that chain. And if the problem is still there at the end, that becomes the rationale for the next call.
Finland is a fun case. The Research Council's fellowship review form gives numerical scores for scientific quality, novelty, feasibility and so on. Societal impact gets no score, and reviewers are told it must not affect the ranking. It can be weighed later as a separate factor. So impact is what the research is sold to the public with, but it's kept out of the scoring. Where someone did check, it wasn't pretty: in one assessment group at STEA (the funder for social and health organisations), only 70 of 184 funded activities had results supported well or sufficiently by evidence.
I'd push "report all meaningful results" one step further. Not every project should promise impact, and basic research shouldn't have to. But a project funded on the promise of reducing a specific problem could file one page up front: baseline, target, who implements, when we'd expect to see it. Then "it didn't move" becomes a reportable result instead of a gap. Same logic as your clinical trials point. You can only report a null if you said what you expected first.
> A “report all results” policy makes the most sense (and would be most easily implemented) where studies have a clear “yes or no” conclusion as to some particular hypothesis.
My impression is that essentially all NIH grants have required hypothesis driven projects for quite some time. With access to the original grants (possibly from FOIA requests?), I expect an LLM could do a pretty good job of determining what hypotheses actually led to publications (usually linked via acknowledgments).
A way to get more negative/null results published is to implement a system in which results do not and cannot determine whether something gets published. Such a system exists: Registered Reports. Stage 1 peer review takes place before the study is conducted and examines the quality of the scientific question and the methodological rigor. At that time, an in-principle decision re publication is made. Then the investigators commence the study, collect data, analyze it, etc. At Stage 2, if the results are null or negative, the reviewers who gave a positive Stage 1 decision cannot change their minds about the quality of the science. More on Registered Reports here: https://www.cos.io/initiatives/registered-reports
I agree with the sentiment. Science would move faster if we could discuss failures more openly and transparently. I often learn what hasn’t worked only through informal, off-the-record conversations at conferences. I suspect we spend considerable time and money repeating versions of experiments that don’t work simply because those results were never reported.
At the same time, your previous post showed that the regulations and reporting requirements tied to NIH funding have steadily increased. Budget limits haven’t kept pace, and there seems to be little serious effort to reduce the reporting burden. Depending on how this is implemented, it could become another substantial, unfunded requirement.
Could we consider a rule that adding one new reporting requirement means removing two existing ones?
Of course in the time between the workshop and the paper there was a radical change in the US presidency to an administration that has been both unscientific and anti-scientific in its approach to research and researchers. Neither the White House nor the present NIH leadership can properly be considered good faith actors and it is therefore difficult to take at face value any proposals they might have for the integrity of scholarly communication. I was surprised that your post makes no mention of the troubled US research landscape. Is it now just taken as read?
This was a great read, and the move to scope the mandate around studies with a prespecified yes-or-no question is what makes it workable rather than the bureaucratic monster you warn about.
The Turner antidepressant numbers you cite carry a second lesson that strengthens your preregistration proposal. Of the 36 negative or questionable trials, only 3 were published in line with the FDA's verdict, but 11 were published looking positive. So the distortion was not only missing nulls; it was nulls rewritten as wins. A results-reporting rule on its own fixes the first problem. Your requirement to preserve the original registration and flag post-hoc changes is what fixes the second, because it gives any reader a fixed point to check the published outcome against.
That ties neatly into your closing point about AI tools. Filling in the missing denominator helps, but a synthesis tool reading spun abstracts would still be misled unless it can see the registered primary outcome sitting next to the reported one. The linkage might matter as much as the disclosure.
One version of this that rarely gets discussed: null impact.
Funders increasingly ask for societal impact, and projects report it. But the reporting usually has no baseline, so it can't come out null. A paper becomes increased knowledge, knowledge becomes deeper understanding, understanding becomes public discussion, and discussion gets written up as impact. Nothing can fail that chain. And if the problem is still there at the end, that becomes the rationale for the next call.
Finland is a fun case. The Research Council's fellowship review form gives numerical scores for scientific quality, novelty, feasibility and so on. Societal impact gets no score, and reviewers are told it must not affect the ranking. It can be weighed later as a separate factor. So impact is what the research is sold to the public with, but it's kept out of the scoring. Where someone did check, it wasn't pretty: in one assessment group at STEA (the funder for social and health organisations), only 70 of 184 funded activities had results supported well or sufficiently by evidence.
I'd push "report all meaningful results" one step further. Not every project should promise impact, and basic research shouldn't have to. But a project funded on the promise of reducing a specific problem could file one page up front: baseline, target, who implements, when we'd expect to see it. Then "it didn't move" becomes a reportable result instead of a gap. Same logic as your clinical trials point. You can only report a null if you said what you expected first.
> A “report all results” policy makes the most sense (and would be most easily implemented) where studies have a clear “yes or no” conclusion as to some particular hypothesis.
My impression is that essentially all NIH grants have required hypothesis driven projects for quite some time. With access to the original grants (possibly from FOIA requests?), I expect an LLM could do a pretty good job of determining what hypotheses actually led to publications (usually linked via acknowledgments).
A way to get more negative/null results published is to implement a system in which results do not and cannot determine whether something gets published. Such a system exists: Registered Reports. Stage 1 peer review takes place before the study is conducted and examines the quality of the scientific question and the methodological rigor. At that time, an in-principle decision re publication is made. Then the investigators commence the study, collect data, analyze it, etc. At Stage 2, if the results are null or negative, the reviewers who gave a positive Stage 1 decision cannot change their minds about the quality of the science. More on Registered Reports here: https://www.cos.io/initiatives/registered-reports
I agree with the sentiment. Science would move faster if we could discuss failures more openly and transparently. I often learn what hasn’t worked only through informal, off-the-record conversations at conferences. I suspect we spend considerable time and money repeating versions of experiments that don’t work simply because those results were never reported.
At the same time, your previous post showed that the regulations and reporting requirements tied to NIH funding have steadily increased. Budget limits haven’t kept pace, and there seems to be little serious effort to reduce the reporting burden. Depending on how this is implemented, it could become another substantial, unfunded requirement.
Could we consider a rule that adding one new reporting requirement means removing two existing ones?
You may be interested to know that a similar proposal emerged from a workshop organised and hosted by NINDS in May 2024 (https://www.ninds.nih.gov/news-events/events/novel-approaches-preventing-publication-bias-workshop) which resulted in this PLOS Biology paper last year: https://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.3003368
Of course in the time between the workshop and the paper there was a radical change in the US presidency to an administration that has been both unscientific and anti-scientific in its approach to research and researchers. Neither the White House nor the present NIH leadership can properly be considered good faith actors and it is therefore difficult to take at face value any proposals they might have for the integrity of scholarly communication. I was surprised that your post makes no mention of the troubled US research landscape. Is it now just taken as read?
This was a great read, and the move to scope the mandate around studies with a prespecified yes-or-no question is what makes it workable rather than the bureaucratic monster you warn about.
The Turner antidepressant numbers you cite carry a second lesson that strengthens your preregistration proposal. Of the 36 negative or questionable trials, only 3 were published in line with the FDA's verdict, but 11 were published looking positive. So the distortion was not only missing nulls; it was nulls rewritten as wins. A results-reporting rule on its own fixes the first problem. Your requirement to preserve the original registration and flag post-hoc changes is what fixes the second, because it gives any reader a fixed point to check the published outcome against.
That ties neatly into your closing point about AI tools. Filling in the missing denominator helps, but a synthesis tool reading spun abstracts would still be misled unless it can see the registered primary outcome sitting next to the reported one. The linkage might matter as much as the disclosure.