The frontiers of biostatistics is filled with a variety of "variance explained" equivalent measures but based on interventional prediction rather than associational/observational prediction (per Pearl's ladder of different "predictions"). They haven't been written up for a mainstream audience. Former RAND statistics group members have done work on this!
This essay made me think of an expectations-feedback loop: individually rational behavior can create collectively irrational science, which is precisely why your micro/macro split feels essential.
Related: Schweiger's Nature piece on the "Szilard point" (where the total cost of applying/reviewing/admin can approach or exceed the value of the funding itself) is a brutal quantification of the macro-incentive distortion you're describing. https://www.nature.com/articles/d41586-025-04060-x
And at the micro level, Panin et al., Time for Tea is a neat analogue: a small mis-specification in the question (present equivalents) can turn into a systematic distortion of the answer (utility curvature masquerading as patience), and their "risk equivalents" fix is basically measurement-level reform. https://doi.org/10.1016/j.jdeveco.2024.103261
I am also cautiously optimistic about the changes in how peer review will be deployed, both at NSF and NIH. Peer review at the agencies was not conceived or designed to make fine-grained distinctions between a set of meritorious proposals. It was meant as a bulwark against political interference into free inquiry (seems relevant), and to advise (not dictate) the decisions of agency employees who were, to quote Vannevar, "persons of broad interest in and understanding of the peculiarities of scientific research and education". At NIH, there are reasons to believe that moving away from strict adherence to pay-by-score-order, which only developed and calcified in recent decades, will have positive effects on support for innovative proposals, emerging areas of research, and early stage investigators. It's something I will be watching closely in the coming months, and I'm sure you will be too.
I think you're naming something we keep forgetting: peer review at agencies wasn't built to be a precision-ranking machine. It's a protective membrane—a bulwark against politics and a way to signal to program officers without handcuffing judgment.
Where it gets weird (and where my "peer review as an industry" alarm goes off) is when that membrane quietly turns into an algorithm: pay-by-score-order becomes the equivalent of "radio chart position," and suddenly everyone learns to write to the chart. That doesn't require bad actors — it's just what systems do when they calcify under load. Standards become expectations become assumptions, and then the pool gets shallower because it's safer to be legible than to be genuinely uncertain.
In my head, NIH/NSF peer review is a bit like the old music ecosystem: DJs originally curated to protect quality and audiences. The incentives (and scarcity) made curation easy to exploit and hard to deviate from. When the metric becomes the product, you don't just select the "best" science — you choose the best-scoring science.
So I'm with you in being cautiously optimistic about moving away from strict score-order. If peer review returns to being a threshold + advice system (not a fine-grained sorting hat), it could create more real space for early-stage folks and for ideas that don't yet know how to "sell a conclusion." I'll be watching too.
The frontiers of biostatistics is filled with a variety of "variance explained" equivalent measures but based on interventional prediction rather than associational/observational prediction (per Pearl's ladder of different "predictions"). They haven't been written up for a mainstream audience. Former RAND statistics group members have done work on this!
This essay made me think of an expectations-feedback loop: individually rational behavior can create collectively irrational science, which is precisely why your micro/macro split feels essential.
Related: Schweiger's Nature piece on the "Szilard point" (where the total cost of applying/reviewing/admin can approach or exceed the value of the funding itself) is a brutal quantification of the macro-incentive distortion you're describing. https://www.nature.com/articles/d41586-025-04060-x
And at the micro level, Panin et al., Time for Tea is a neat analogue: a small mis-specification in the question (present equivalents) can turn into a systematic distortion of the answer (utility curvature masquerading as patience), and their "risk equivalents" fix is basically measurement-level reform. https://doi.org/10.1016/j.jdeveco.2024.103261
(If useful, I also found Hazée et al.'s open science effectiveness + adoption audit a convenient "full-stack" bridge: https://doi.org/10.1177/10946705251338461.)
I am also cautiously optimistic about the changes in how peer review will be deployed, both at NSF and NIH. Peer review at the agencies was not conceived or designed to make fine-grained distinctions between a set of meritorious proposals. It was meant as a bulwark against political interference into free inquiry (seems relevant), and to advise (not dictate) the decisions of agency employees who were, to quote Vannevar, "persons of broad interest in and understanding of the peculiarities of scientific research and education". At NIH, there are reasons to believe that moving away from strict adherence to pay-by-score-order, which only developed and calcified in recent decades, will have positive effects on support for innovative proposals, emerging areas of research, and early stage investigators. It's something I will be watching closely in the coming months, and I'm sure you will be too.
I think you're naming something we keep forgetting: peer review at agencies wasn't built to be a precision-ranking machine. It's a protective membrane—a bulwark against politics and a way to signal to program officers without handcuffing judgment.
Where it gets weird (and where my "peer review as an industry" alarm goes off) is when that membrane quietly turns into an algorithm: pay-by-score-order becomes the equivalent of "radio chart position," and suddenly everyone learns to write to the chart. That doesn't require bad actors — it's just what systems do when they calcify under load. Standards become expectations become assumptions, and then the pool gets shallower because it's safer to be legible than to be genuinely uncertain.
In my head, NIH/NSF peer review is a bit like the old music ecosystem: DJs originally curated to protect quality and audiences. The incentives (and scarcity) made curation easy to exploit and hard to deviate from. When the metric becomes the product, you don't just select the "best" science — you choose the best-scoring science.
So I'm with you in being cautiously optimistic about moving away from strict score-order. If peer review returns to being a threshold + advice system (not a fine-grained sorting hat), it could create more real space for early-stage folks and for ideas that don't yet know how to "sell a conclusion." I'll be watching too.
Very interesting article in Asterisk. Relevant to clinical trials