This essay is a personal history of the $60+ million I allocated to metascience starting in 2012 while working for the Arnold Foundation (now Arnold Ventures).
Brilliant piece. I've never, as a working scientist, understood the whole resistance to replication 'mindset' (a phrase I use colloquially 😉, given recent controversies). The greatest thrill you can have is that someone else gets the same result as you did: it means you have either discovered something about the world that is independent of you, or you've both been confounded by the same artifact. One hopes for the former, but it can be the latter. It also means you've potentially discovered something that others regard as worth probing further. Amazing and wonderful venture: congratulations.
The most important philanthropy of the 21st century - if we can't trust science, everything else will collapse. And existing peer review, prestige systems, and funding agencies were not addressing these issues.
Thanks for this excellent post! Some miscellaneous thoughts:
1. The pyramid is great, and I plan to annoy management at my institute (and maybe even at funding agencies!) with it for many years to come. Inverting the pyramid is an epidemic among bureaucrats.
2. I would be curious to see your perspective on the Nutrition Science Initiative and the struggles of nutrition science more broadly. I'm usually quite critical of nutrition science, the whole field of which seems to be a textbook case of trying to examine small, confounded effects with inadequate sample sizes and insufficient controls.
3. As a practicing scientist, this line from the Open Science Framework:
> allowing open, continuous peer review
might be a contender for my new favorite definition of hell. Peer review is often worthless (see e.g. https://www.experimental-history.com/p/the-rise-and-fall-of-peer-review); I can personally think of a few cases where a referee has improved one of my papers, but this is so rare I can count the cases on one hand. I'm not sure what continuous peer review is supposed to accomplish besides making everyone hate each other as well as their own research (projects need to end eventually). If the actual goal is for scientists to give worthwhile feedback on ideas even after they've been published, I think that should happen through subsequent publications, i.e. via citations. Technology could help assign valence to citations, since a paper can also be cited by someone rebutting it. (We might also need a category of neutral citations, i.e. the usually irrelevant "related research" citations in the introduction that are included for political reasons.) Otherwise, I don't really see the need for a new system here; we should just make better, more effective use of the system we already have. (See also the freak takes article on how journal publication used to be just one step in evaluating an idea: https://www.freaktakes.com/p/feynman-on-journal-reviews-conferences).
This article is in total denial of the ugly side of the Open Science Movement. I was working primarily in health psychology/ behavioral health and witnessed the savaging of those of who of us made early requestes for release of data. Large behavioral trials with long follow ups cannot wait for replication. Armed with Koch money, which Stuart may have shared, Brian Nosek engaging in bullying those of us who saw him as needlessly bureaucratizing research when speedy removal of bad science from the literature would have been much more efficient and effective.
Brian did not support release of the PACE trial data, even though the investigators had promised to make it available as a condition of publishing in PLOS One.
The erasure of the PACE trial from Stuart seriously damages the credibility of his history.
Pre-registration was already a failure in behavioral medicine before Nosek promoted it in psychology.
I read this while supporting researchers working on the slow crisis of care, and what they kept saying was not that the science was hard. It was that they could not do the research and do the caring, and one of the two was going to lose. Mostly women, mostly mid-career, often people whose grant paperwork crosses my desk.
Two things first. This is the most useful account of research-reform funding I have read. It is useful because it is specific: the figures, the failures left in — Open Trials stalling, the tDCS grant that produced nothing, the three groups you say you should not have tried to patch together, and the moment you stop to say that Johanna Cohoon and Mallory Kidwell deserved more recognition than they got. Retrospectives written from that altitude usually do none of that.
Your talent heuristic, which you state twice, is to find smart people already doing great things in their spare time and empower them with additional capital. The funding step is right. Goldacre to Oxford, Chambers bought out of Cardiff, unpaid hours converted into paid ones. But a detection step runs before the funding step, and it tests for spare time. Call it the spare-time test, which money can buy, but it has to be there already. Being seen costs hours, and hours are the first thing care takes. The people I am describing fail it before anyone reaches their ideas.
I think this is structural rather than a matter of anyone's attention, and there is a case where you can watch it get measured. Brainport Eindhoven is Europe's model triple-helix region, the one everybody cites, born from crisis and told very cleanly by the top management. The Dutch government committed €2.5 billion through Operation Beethoven to maintain the industrial layer. Then in December 2025 Wilthagen and Meijdam at Tilburg, working with RaboResearch and Utrecht, published World Class Industry, World Class Society, which found that about 13 percent of the region's residents — roughly 100,000 people — score poorly across multiple broad-prosperity measures, and asked the question that keeps me, a working mom/caretaker supporting some amazing researchers in similar and sometimes obviously more stressful conditions: why is there an Operation Beethoven for industrial scaling and not much that is equivalent for education, health and social cohesion? In fairness to the Dutch region, it is moving; 5,400 student homes over eight years were announced in June. The asymmetry is still the finding, and the useful thing about it is that it is a number rather than a mood. The USA, however, is a completely different place, and I’d argue it lacks a lot of what Europe has, even if it could do better.
You say twice, honestly, that your categories here are retrofitted. Ada Palmer's Inventing the Renaissance is a book about what retrofitting does — a golden age is assembled afterward from whatever survived, and what survives is correspondence between people who were already legible to one another, who, in Eindhoven’s words, ‘mattered’, who already had money and spare time. Your sources are your Gmail, old Google Docs, links recovered from the Internet Archive. That is a real archive, and it is a funder's archive. Nobody keeps the coordinator's inbox, and I think this means we're missing out on funding some really great stuff. I’m not sure how to address this, but I read here for ideas and opportunities.
Holy shit, man, you’re my hero and I didn’t even know it! I mean, you funded all my heroes and their most heroic work. So that makes you…Zeus? Not sure but holy hell thank you. And just flippantly mentioning, “oh yeah, and I funded Prasad’s Ending Medical Reversals.” You launched Prasad?? Damn. Thank you from the bottom of my heart. My career in the biosciences has meaning because of your work.
As the co-founder of the Reproducibility Initiative with Elizabeth, this matches my recollection as well. One note - Elizabeth and I were funded by Arnold & COS was our fiscal sponsor. We ran the reproducibility project, managed all the replications, did the media outreach, and met with journals and funders about policy implications. We hired Tim as a project manager to coordinate posting of replication studies on the OSF and he was a COS employee due to this arrangement. I'm pleased with how it helped develop his career.
Brilliant piece. I've never, as a working scientist, understood the whole resistance to replication 'mindset' (a phrase I use colloquially 😉, given recent controversies). The greatest thrill you can have is that someone else gets the same result as you did: it means you have either discovered something about the world that is independent of you, or you've both been confounded by the same artifact. One hopes for the former, but it can be the latter. It also means you've potentially discovered something that others regard as worth probing further. Amazing and wonderful venture: congratulations.
What a run!
Maybe for a future post... how would you train or advise a group of new metascience VCs? What advice would you give?
Good question! Let's catch up by phone soon?
The most important philanthropy of the 21st century - if we can't trust science, everything else will collapse. And existing peer review, prestige systems, and funding agencies were not addressing these issues.
Thanks for this excellent post! Some miscellaneous thoughts:
1. The pyramid is great, and I plan to annoy management at my institute (and maybe even at funding agencies!) with it for many years to come. Inverting the pyramid is an epidemic among bureaucrats.
2. I would be curious to see your perspective on the Nutrition Science Initiative and the struggles of nutrition science more broadly. I'm usually quite critical of nutrition science, the whole field of which seems to be a textbook case of trying to examine small, confounded effects with inadequate sample sizes and insufficient controls.
3. As a practicing scientist, this line from the Open Science Framework:
> allowing open, continuous peer review
might be a contender for my new favorite definition of hell. Peer review is often worthless (see e.g. https://www.experimental-history.com/p/the-rise-and-fall-of-peer-review); I can personally think of a few cases where a referee has improved one of my papers, but this is so rare I can count the cases on one hand. I'm not sure what continuous peer review is supposed to accomplish besides making everyone hate each other as well as their own research (projects need to end eventually). If the actual goal is for scientists to give worthwhile feedback on ideas even after they've been published, I think that should happen through subsequent publications, i.e. via citations. Technology could help assign valence to citations, since a paper can also be cited by someone rebutting it. (We might also need a category of neutral citations, i.e. the usually irrelevant "related research" citations in the introduction that are included for political reasons.) Otherwise, I don't really see the need for a new system here; we should just make better, more effective use of the system we already have. (See also the freak takes article on how journal publication used to be just one step in evaluating an idea: https://www.freaktakes.com/p/feynman-on-journal-reviews-conferences).
3.
This article is in total denial of the ugly side of the Open Science Movement. I was working primarily in health psychology/ behavioral health and witnessed the savaging of those of who of us made early requestes for release of data. Large behavioral trials with long follow ups cannot wait for replication. Armed with Koch money, which Stuart may have shared, Brian Nosek engaging in bullying those of us who saw him as needlessly bureaucratizing research when speedy removal of bad science from the literature would have been much more efficient and effective.
Brian did not support release of the PACE trial data, even though the investigators had promised to make it available as a condition of publishing in PLOS One.
The erasure of the PACE trial from Stuart seriously damages the credibility of his history.
Pre-registration was already a failure in behavioral medicine before Nosek promoted it in psychology.
See "Replication initiatives will not salvage the trustworthiness of psychology" https://bmcpsychology.biomedcentral.com/articles/10.1186/s40359-016-0134-3
I read this while supporting researchers working on the slow crisis of care, and what they kept saying was not that the science was hard. It was that they could not do the research and do the caring, and one of the two was going to lose. Mostly women, mostly mid-career, often people whose grant paperwork crosses my desk.
Two things first. This is the most useful account of research-reform funding I have read. It is useful because it is specific: the figures, the failures left in — Open Trials stalling, the tDCS grant that produced nothing, the three groups you say you should not have tried to patch together, and the moment you stop to say that Johanna Cohoon and Mallory Kidwell deserved more recognition than they got. Retrospectives written from that altitude usually do none of that.
Your talent heuristic, which you state twice, is to find smart people already doing great things in their spare time and empower them with additional capital. The funding step is right. Goldacre to Oxford, Chambers bought out of Cardiff, unpaid hours converted into paid ones. But a detection step runs before the funding step, and it tests for spare time. Call it the spare-time test, which money can buy, but it has to be there already. Being seen costs hours, and hours are the first thing care takes. The people I am describing fail it before anyone reaches their ideas.
I think this is structural rather than a matter of anyone's attention, and there is a case where you can watch it get measured. Brainport Eindhoven is Europe's model triple-helix region, the one everybody cites, born from crisis and told very cleanly by the top management. The Dutch government committed €2.5 billion through Operation Beethoven to maintain the industrial layer. Then in December 2025 Wilthagen and Meijdam at Tilburg, working with RaboResearch and Utrecht, published World Class Industry, World Class Society, which found that about 13 percent of the region's residents — roughly 100,000 people — score poorly across multiple broad-prosperity measures, and asked the question that keeps me, a working mom/caretaker supporting some amazing researchers in similar and sometimes obviously more stressful conditions: why is there an Operation Beethoven for industrial scaling and not much that is equivalent for education, health and social cohesion? In fairness to the Dutch region, it is moving; 5,400 student homes over eight years were announced in June. The asymmetry is still the finding, and the useful thing about it is that it is a number rather than a mood. The USA, however, is a completely different place, and I’d argue it lacks a lot of what Europe has, even if it could do better.
You say twice, honestly, that your categories here are retrofitted. Ada Palmer's Inventing the Renaissance is a book about what retrofitting does — a golden age is assembled afterward from whatever survived, and what survives is correspondence between people who were already legible to one another, who, in Eindhoven’s words, ‘mattered’, who already had money and spare time. Your sources are your Gmail, old Google Docs, links recovered from the Internet Archive. That is a real archive, and it is a funder's archive. Nobody keeps the coordinator's inbox, and I think this means we're missing out on funding some really great stuff. I’m not sure how to address this, but I read here for ideas and opportunities.
Holy shit, man, you’re my hero and I didn’t even know it! I mean, you funded all my heroes and their most heroic work. So that makes you…Zeus? Not sure but holy hell thank you. And just flippantly mentioning, “oh yeah, and I funded Prasad’s Ending Medical Reversals.” You launched Prasad?? Damn. Thank you from the bottom of my heart. My career in the biosciences has meaning because of your work.
As the co-founder of the Reproducibility Initiative with Elizabeth, this matches my recollection as well. One note - Elizabeth and I were funded by Arnold & COS was our fiscal sponsor. We ran the reproducibility project, managed all the replications, did the media outreach, and met with journals and funders about policy implications. We hired Tim as a project manager to coordinate posting of replication studies on the OSF and he was a COS employee due to this arrangement. I'm pleased with how it helped develop his career.