Picking winners or hedging bets?

Federal science agencies have long wrestled with the question of how to ensure their billions of dollars of grants and contracts result in maximum benefits for Americans.

On the surface, it might seem that a goal-oriented, interventionist style of management would be the best policy: prioritize our most pressing problems and distribute funding accordingly.This top-down strategy creates a unique hazard, though: bad choices can send resources in the wrong direction, resulting in slow progress that impedes the delivery of tangible benefits to citizens.

Moreover, if decision makers pursue an applied advance when the underlying scientific principles remain poorly understood, they risk costly failures and a loss of support for the research enterprise at large. Examples include Nixon’s 1971 War on Cancer, undertaken on the premise that a cure could be achieved in less than a decade, the drive to build an operational commercial fusion plant by the mid-1990s, and the National Plan to Address Alzheimer’s Disease, which failed to produce the effective treatments and prevention strategies promised by 2025.

The contrasting laissez-faire approach is to hedge your bets. If the path to an advance is impossible to predict, then the wisest choice might be to distribute funding to as widely as possible. The main risk of this strategy is that spreading funding thinly may limit the resources available to the most promising research, resulting in the same negative outcomes as the interventionist approach: limited support for the most promising topics, slow progress, and delayed returns.

In reality, at science agencies like the National Science Foundation and National Institutes of Health, program managers navigate between these two extremes, setting some funds aside to target high-priority areas and using the rest to cover the widest possible range of meritorious proposals. This middle path seems like the most rational way to proceed, but it still requires making choices. How much should be set aside for high priority topics? What should those topics be? How should we define merit? Expert opinion can provide a helpful guide, but leading experts can have remarkably different opinions about what matters and what should be prioritized. Moreover, experts and administrators alike can be risk averse, giving an edge to high-profile, well-established concepts.

What’s really needed is a data-driven framework to help guide decision-making, one that doesn’t simply repackage prestige or incumbency.

Recognizing breakthroughs years in advance

Many breakthroughs fail to attract funding or recognition at the earliest stages of development. Katalin Karikó, for example, spent years struggling to win grants for work on mRNA that would eventually win her and Drew Weissman a Nobel Prize. Similar stories can be told about Jim Allison’s groundbreaking studies leading to the development of cancer immunotherapy, Stan Pruisner’s demonstration that prions can self-replicate without DNA, or Robin Warren and Barry Marshall’s proof that Helicobacter pylori, not stress, causes ulcers.

Yet at some point, the scientific community recognizes and responds to transformative breakthroughs like these. How does that change happen? And what if science funders could identify the areas that are on the cusp of a breakthrough?

Those questions led my collaborators and me to study how scientists responded to past breakthroughs. Based on patterns in that data, we developed a means of predicting future breakthroughs, using biomedicine as a proof of concept.

It’s worth taking a moment to describe how our approach works. Briefly, our prediction process begins by grouping the 18 million or so peer-reviewed papers in PubMed, the authoritative database of biomedical research, into roughly 50,000 unique topics. A small subset of these topics are so widely acclaimed as breakthroughs that they have been recognized with a major prize like a Nobel or a Lasker Award. We theorized that studying the development of these outliers over time, especially before they gained acclaim, was the key to identifying future instances of them.

After defining topics, we chose 21 representative breakthroughs and asked what they looked like before they won prestigious prizes. To answer that question, we re-ran our topic-mapping algorithm repeatedly, winding back the clock one year each time, stringing together the years to follow the progress of each field as it grew and advanced.

We found that in each case, the years leading up to a discovery followed a pattern: a burst of papers on a new, rapidly evolving topic, many of which quickly became influential. These characteristics—the percent of papers that are new, the percentage of papers that are brought in from other topics, and the influence of each individual paper—can all be measured separately for any given topic. Together, they act as a predictive signal that is detectable an average of five years before a transformative discovery is made, and up to thirty years before the discovery receives a major prize. Scanning the current research landscape for topics that display this signal allows us to predict what work will likely produce a future breakthrough. Each year we analyzed includes four or five signals, a number that appears to have remained constant across a twenty year time frame.

In their classic work on the sociology of science, Bruno Latour and Steve Woolgar established that the desire to participate in a major discovery is an important motivator for scientists as they consider whether they should take up (or abandon) a research problem. Although further work is needed, the simplest explanation for our results is that this behavior is widespread enough to detect at scale. A foundational advance, or something like it, draws the attention of scientists and causes them to change the direction of their research. They vote with their feet, staking their own reputation and careers by publishing on and citing the new idea. That upswell of interest is the foundation of our breakthrough signal.

What kind of research does our model identify as a breakthrough?

After building a model based on the patterns in our 21 preselected breakthroughs, we tested it by identifying all early signals of discovery from 1994 through 1997. Of the 18 signals we found in that time period, 17 can be traced forward in time to a breakthrough. Examples include super-resolution fluorescence microscopy (see the image above), the directed evolution of proteins and enzymes (Chemistry Nobel, 2018), and sequencing the human genome (National Medal of Science, 2008).

The discovery of the role of leptin signaling in obesity is a particularly interesting case study. Around 1950, scientists noticed the existence of a type of lab mouse that suffered from a unique inherited form of obesity. When given the freedom to choose their own diet, these mice ate more than normal and preferred food that was high in fat. Researchers had few clues as to why the animals overate; early experiments pointed to the existence of a soluble ob factor found circulating in the bloodstream that regulated appetite, but its identity was unknown.

In 1994, Jeffrey Friedman and colleagues identified and sequenced the ob gene, noting that the protein it encoded appeared to fit the profile of a circulating factor. A year later, a trio of high-profile publications demonstrated that ob was a hormone that regulated body weight and fat deposition by regulating appetite. One of these was co-authored by Friedman, who christened it leptin from the ancient Greek word for thin. Using our model, the data from 1996 make it clear this advance would become a breakthrough. Scientists had flocked to study the biology of the new hormone, generating the flurry of new papers and citations required to produce a signal. One of these demonstrated that serum leptin concentrations reflected the amount of adipose tissue in the human body, providing clinical validation of the earlier mouse studies. Before the identification of leptin, scientists were unable to point to a specific molecule that controlled appetite and adiposity, so that it was impossible to rule out non-physiological causes of obesity; its characterization fundamentally changed our concept of weight gain. In 2010, Friedman and fellow pioneer Douglas Coleman were awarded a Lasker for the discovery.

Our work also identifies breakthroughs in clinical practice and behavioral and social sciences, even though these areas are less likely to draw the attention of major prize committees. One example that falls into this category is the introduction of endovascular aneurysm repair, a minimally invasive surgical procedure for the treatment of abdominal aortic aneurysm, a leading cause of death among Americans over the age of 65. Relative to the previous standard of care, it reduces in-hospital mortality by almost four-fold. In 2017, Timothy Chuter received the American College of Surgeons’ prestigious Jacobsen Innovation award for his multiple refinements to the technique.

Another under-recognized breakthrough identified by our model is the development of standardized survey instruments to assess the quality of life for HIV⁺ patients. The antiretroviral cocktails introduced in the mid-1990s reduced AIDS mortality, but treatment came with serious side effects. The new evaluations showed that patients whose disease progressed had worse physical functioning than Americans with other chronic diseases, while those who were asymptomatic maintained a health-related quality of life on par with the overall US population. This evidence made it possible for patients and activists to argue that even in the absence of a total cure, and in spite of the side effects, alleviation of symptoms was a worthwhile priority for clinicians. My colleagues and I have failed to identify any significant recognition for the physicians and health policy experts who pioneered this advance, although in 2016, researchers called for the World Health Organization to set targets for good health-related quality of life as part of its framework to end the AIDS pandemic.

From forecasting to funding

How should science funders use our predictions?

Any serious discussion of what we should do with this new capability needs to begin with an understanding of its limitations. We can’t (and shouldn’t) invest all of our resources in the small number of topics predicted to produce breakthroughs. Innovative new fields are born from existing ones; they rarely arise de novo. If funders don’t maintain a diverse portfolio, the breakthroughs of tomorrow will have no antecedents. Think of this as eating the seed corn, or killing the goose that lays the golden egg.

There are also multiple goals of funding beyond basic scientific discovery, including translating discoveries into clinical practice, developing new technologies, supporting economic development, and training new scientists. The investments needed to accomplish these goals are likely different from those required to support emerging breakthroughs.

Finally, we must be aware that we may not capture every instance of a breakthrough. The concept is fuzzy, and not every significant discovery is recognized with a major award. Further, breakthroughs are rare, meaning that our dataset is, by necessity, small. This makes formal estimates of accuracy challenging.

Although our overall success rate appears to be high, any one signal may turn out to be a false positive. We found one apparent example: the development of the third generation COX-2 inhibitors, Vioxx and Celebrex, which were much celebrated in the early 2000s as effective but non-addictive pain relievers, meets all the algorithmic criteria of a breakthrough. Five years after it received FDA approval, researchers demonstrated that Vioxx was associated with serious cardiovascular complications, and the manufacturer voluntarily pulled the drug from the market. Celebrex remains available, although like other NSAIDs, it carries an FDA warning for increased cardiovascular risk. Decision makers who were overly focused on the breakthrough potential of this research might have wasted substantial resources nurturing an area that ultimately proved to be a failure—but so would those who relied on expert opinion at the time.

That said, the potential benefits of funding the next mRNA vaccine or cancer immunotherapy years earlier than we might have are large. When we can’t assume the outcome of an action, it’s worth conducting an experiment. Can targeted support for areas that are poised to produce a breakthrough increase the return on scientific investment without cannibalizing future advances?

With the necessary administrative infrastructure in place, conducting this experiment would be straightforward. First, identify all the breakthrough signals between 2018 and 2023, select half at random, and commit to a decade of investment. For biomedicine, a six year window should yield about 30 topics with the potential to produce a breakthrough. Providing strong support to half of these might be accomplished for $300 million a year. That’s roughly half of the recent annual budget of the NIH Common Fund or around 20% of the fiscal year 2025 budget for ARPA-H. After five years, then again at ten, compare to see which group — the one that was actively managed, or the control — produced more breakthroughs, and on what timescale. Although we used biomedicine as an example, the same experiment could just as well be run for physics, materials science, or any other discipline.

Any attempt to undertake such an experiment should keep three points top of mind. First, we can predict the topic of future breakthroughs, but the people and vision for how to move the work forward are still important. In practice, this means success requires active program managers with the expertise to develop the science and the authority to do so, including by recruiting investigators. Second, we would be wise to learn from the failures of past attempts to fund transformative research. High among these are funding modestly repackaged work, confusing the genuinely novel with the merely unfamiliar, and relying too much on incumbent investigators. All of these risks can be mitigated by thoroughly and intelligently integrating high quality data into portfolio management. Finally, we need to have patience. Even with strong funding support, producing a breakthrough takes time.

If supporting breakthroughs is the best way to nurture the birth of new fields, we also need to accurately describe and properly manage the other two stages, so that we invest wisely throughout the productive lifetime of ideas and divest once they reach the point of diminishing returns. We would benefit from more study of all of these phases, especially the transition from one to another. More research could answer questions like:

• When does a breakthrough become an established field?

• How do the availability of funding, the size and characteristics of the available workforce, and the accumulation of evidence that disagrees with prevailing models affect a field’s decline?

• Exactly how do old ideas give birth to new ones, and can we speed up the process?

The more we understand about the dynamics of scientific progress, the better we can maximize the odds that our curiosity leads to results. I’d call that finding a way to make our own luck.

The bill itself doesn’t say a lot that would be relevant to metascience, with two possible exceptions.

Section 214 of the bill says that the NIH Director can enter into “other transactions” (i.e., a flexible term referring to something other than the usual grants and contracts) to do research under Section 402(b)(12) of the Public Health Service Act.

The section says the NIH Director can fund the following stuff:

The budget bill then says that the NIH Director doesn’t have to use the normal peer review process in such cases:

Finally, the NIH Director has to notify Congress if he spends more than $100m on any one transaction:

To be honest, I’m not sure what Congress or the NIH is contemplating here, but it sounds interesting.

Next, Section 223 of the budget says that NIH can’t spend any money on indirect costs above “30 percent of an award” going to privileged universities:

Section 4968 of the Internal Revenue Code is about taxing universities with huge . . . tracts of land (or, more broadly, endowments).

It’s not clear how much this requirement would even matter, given that it defines indirect costs as a percentage of the award, not as a percentage of the direct costs (which is the usual, albeit misleading, formulation). And given that effective indirect cost rates are already in that neighborhood, a 30% ceiling might not matter even for the wealthiest universities.

Anyway, on to the report language accompanying the bill, where there are a number of instructions that might affect the quality of the science we’re funding.

First, there is the issue of clinical trials that involve international partners. While we might occasionally need more oversight and control over such partnerships, they are often immensely valuable. The House recognizes that tradeoff:

Second, a very different issue is how to provide AI models with access to a secure but useful data platform, NIH should be much more proactive in this direction:

Third, there’s the issue of how to encourage more diversity amongst the institutions that receive federal funding:

Fourth, the infamous indirect cost issue. This is still a lingering cause for many people, and the answer is yet to be determined. The House weighs in as follows:

Fifth, the issue of administrative burden, about which we have written many times.

Sixth, the issue of replication experiments and fraud detection, which remains important:

All in all, it is good to see that the House Appropriations Committee is engaging with metascience issues coming from places like the Good Science Project and others.

The bottleneck

At this point, it is uncontroversial to say that science needs to stop using the PDF article as the unit of knowledge and currency. The unbearable slowness of scientific publishing, the profit motive and margins: I’m not saying anything new. The PDF also sucks because it’s hard to extract structured information, which makes it hard to do evidence synthesis.

As a result, we do much less evidence synthesis than is needed. And evidence synthesis ultimately undergirds most policy and medical decision-making. I can see second-by-second real-time odds for any sporting or newsworthy event, but a school board can’t see the best evidence on whether their 8th graders should be taught algebra.

As a society, we don’t treat this as an important problem. Again, not controversial.

Not only is the problem well understood, but the solution has already been laid out. What we need is AI-assisted living evidence synthesis - (1) an open knowledge graph of atomic claims, (2) claims linked to evidence, (3) assessment and synthesis of each piece of evidence, and (4) continuous updating with new data.

What few realize (yet) is that the technical capacity to build this vision for a significant portion of science already exists. Not only that, scientists and startup teams are already building many of these components. I know this because I’ve been surveying the space and talking to many of the builders. There are some missing pieces: for example, evaluations of how well some of the components work. But at this point, most of what’s missing is a fully end-to-end working integration of all of these parts.

In the rest of this essay, I’m going to lay out all the parts of a working living evidence layer for science and who is working on them, and propose concrete next steps for building this system.

What’s now possible

The diagram above outlines the components of a living evidence synthesis platform, including some of the teams working on each component¹. Scientific PDFs are processed into claims with associated evidence. The evidence is subjected to a forensic audit, methodological evaluation and robustness and reproducibility checks. Finally it’s given a weight in a continuously updating synthesis. What follows is a description and status of each component and a few of the teams working on them.

Document understanding

The first thing you need to be able to do is turn an article into structured data. Mostly that means parsing PDFs. There are often multicolumn layouts that confuse non-specialized PDF to text processing libraries. For scientific papers, there is the added complexity of parsing formulas and tables and figures.

This is a really hot area - there are startups offering APIs, and it seems like a new open source package gets posted to Github every few weeks. What follows isn’t exhaustive. A package called GROBID was the state of the art for a while, they didn’t update their package for nearly two years until very recently. In the meantime reducto.ai released an AI powered PDF extraction API, PaddleOCR became popular, IBM released a model called Docling, and both Mistral and Gemini created models and libraries. I also know of at least one other well-funded psychology research group working on a paper parser.

By contrast, there are few open evals in this space, with no extensive evals for complex table comprehension in particular. Nonetheless, I’m confident this will be a solved problem soon, given the combination of LLM advances and developer interest.

Hypothesis level extraction

There has been increasing interest in comprehending the extracted text of papers and linking information to evidence for each hypothesis. A lot of work has already been done. Trialstreamer (Marshall et al. 2020) and RobotReviewer LIVE (Marshall et al. 2023) demonstrated automated extraction of trial population, intervention, and outcome at scale on clinical RCTs. PaperQA2 (Skarlinski et al. 2024) and Ai2 ScholarQA (2024) extended this to retrieval-augmented question answering with citation grounding. Elicit, Consensus, and SciSpace operationalized claim-level extraction for end users. OpenEval (Booeshaghi et al. 2026) is the most recent and most ambitious: 1.96 million atomic claims extracted from 16,087 eLife manuscripts using Claude Sonnet 4.5, grouped into ~299,000 results, with LLM evaluations showing 81% agreement with human peer review on a 2,487-paper subset.

None of these solutions link claims to test-statistics, as you would need to evaluate randomized controlled trials. This is why I built the evidence.guide API - to extract hypotheses and associated test statistics from behavioral science papers. The best public eval of this kind of extraction I’m aware of comes from the recent SCORE project - they had humans code thousands of psychology papers to extract their claims by hand. It would be extremely helpful to the world if all scientific PDFs were available as structured open data.

I’ve been working to make this happen, both directly at Berkeley and through coordination with large entities I can’t yet speak of; as hard as it is to do, I think it’s possible².

Forensic audit

A lot of work has been done on forensic audit, but some gaps remain. Of course, for biology papers that rely on images for evidence, there are a variety of tools (notably Proofig and ImageTwin) to spot anomalies. These are still well short of what sleuths like Elizabeth Bik can do on her own, but these tools are constant companions among fraud analysts. There’s someone working on auditing Excel files for anomalies, and a number of teams are automating numerical checks like GRIM and SPRITE, including Scrutiny project, the INSPECT-SR team as well as statcheck. The regcheck team is building a way to use AI to compare preregistrations to analyses in papers, to ensure there aren’t significant deviations. Nonetheless, there are many other kinds of anomalies to screen for, both public and less publicly known. And there are no formal evals for anomaly detection that I’m aware of. Still, there’s a lot to draw from in this space and I’m pretty certain we will be able to scan papers for most kinds of obvious anomalies in the near future.

Methodological review

This area has been white hot, though I fear for many of the startups in this space, because this capability may become commoditized. There are at least six different AI peer review companies, including Refine.ink, Reviewer3, ReviewerZero.ai, Q.E.D. Science, Paper Wizard, and Isitcredible. Coarse (a pun on refine) was also recently created as an open source alternative. These systems provide qualitative feedback on the content of papers, spotting methodological weaknesses and mathematical errors. They seem to work pretty well, and many academics report bitterly that they exceed the average quality of typical peer reviewers. But there are few evals here either. What evals exist so far involve using LLM-as-judge (circularity problems abound) or comparing against human reviews of questionable quality. What you’d ideally want is an eval that measures capturing known errors in papers³.

Reproducibility and robustness

Another active area has been using AI agents to automate computational reproducibility⁴ and robustness⁵ checks in papers that report numerical results. For more recent papers where data and code are available, AI agents can see whether they can re-run the analyses and produce the numbers reported in the published paper. In addition to a handful of individual academics who have been experimenting using Claude Code for this, the Institute for Replication is a leading group working on building an end to end system. The evals related to this problem are the most mature, with CORE-Bench (Siegel, Kapoor, Narayanan 2024) and PaperBench (OpenAI 2025) available to benchmark agents on this task. There is also work on getting AI agents to test alternative ways of analyzing the data to ensure the results are robust to small analytic design choices.

Synthesis

This is the most underdeveloped area where significant investment is required. Although some automated evidence synthesis systems exist — for example, otto-sr is building an AI agent to write systematic reviews — none of these incorporate the full range of paper level signals to weight evidence appropriately. Nor is there anything like an eval or a gold standard for a good systematic review. Arguably Cochrane reviews are the closest we have to gold standard human systematic reviews, though I’ve heard academics in the know complain about their uneven quality. A key question for a synthesis platform is how to weight anomalies and methodological issues in assessing the quality of a piece of evidence. This is an unsolved problem and one I’m very keen to work on.

Continuous updating

There are many pieces of basic infrastructure available for monitoring for new research and initiating updates. OpenAlex is the current open citation graph. Retraction Watch integrated into Crossref in October 2023. Scite tracks how citations support, contrast, or mention prior claims. The Living Evidence Network demonstrated continuous-update workflows in clinical guidelines. Engineering this is a relatively straightforward task.

When you look over this technical architecture and all the progress being made, it’s hard not to be optimistic that a living guide to scientific evidence will be built.

The stakeholders are ready

The social infrastructure for this is starting to coalesce — it’s not just a pie in the sky academic exercise to imagine this coming into existence. Institutions like the Center for Open Science, the Institute for Replication, the INSPECT-SR, the Living Evidence Network and more are all working on scaling work to improve research quality.

Funders are also aligned. The Sloan Foundation has funded living evidence work through COS. Coefficient Giving supports the Institute for Replication and COS. The Astera Institute and the Institute for Progress have shown interest in this space. NIH has established an Office for Replication and Reproducibility. Although there are (very unfortunately) serious headwinds in science funding generally, there is an active group of funders interested in metascience.

A brief word about what I’ve been doing at RDI. First, as a Visiting Scholar at Berkeley I’ve been actively figuring out how a non-profit and a public university can conduct and make public the results of large-scale academic article data mining. With some of the money I raised from donors, I commissioned a legal analysis of recent case law and publisher text data mining (TDM) agreements in order to understand whether a massive open data mining of academic articles is possible (with caveats, it is). I’ve also been working to bring together stakeholders in this space, and identify gaps. I’ve also been doing some software development in this space, with more to come.

A pilot proposal

The assumption undergirding all of this is that an AI, given all this information, would make the right judgment about a scientific claim with lots of conflicting evidence, weighing all the factors appropriately. That’s the hypothesis we need to test.

Randomized control trial research is the best place to focus on first. RCTs are used to make many of the important decisions in society - from medical trials to public policy changes. And they use a relatively uniform set of inferential statistics with lots of known and available diagnostics. Behavioral science experiments, within the broader realm of RCTs, should be first used as a testbed whose results can be generalized. Because behavioral science is at the vanguard of open science practices, replications abound (there are thousands of them) to serve as ground truth training data.

Key Hypothesis

Therefore the pilot would test, in behavioral RCTs that have been replicated, whether the quality of evidence for a claim can be used to accurately predict whether that claim will replicate.

Secondary Hypotheses

1. Compared to claims that replicated, non-replicated claims demonstrate a greater share of forensic anomalies in their source literatures.

2. Hypothesis level claim and statistic extraction is accurate enough to scale living evidence without onerous human review costs.

3. Replication prediction is more accurate than prediction markets⁶ or journal prestige.

If all the different quality signals we gather do accurately predict which studies will replicate, then we can use that model to score evidence to power the living evidence layer.

Why this is informative regardless of outcome

If the pilot succeeds, the architecture extends to medical RCTs (where Living Evidence already operates and integration is mostly about claim representation), then to slices of basic biology with stable replication structure. If it fails, the field learns which quality signals are load-bearing and which ones metascience has oversold. Either result is a contribution to knowing what the literature supports.

Implications for funders

Because this burgeoning ecosystem of builders already exists, a well-informed philanthropic or government funder could play a crucial catalyzing role in bringing this future about. They could play at least three roles: creating open structured datasets, publishing open benchmarks and incentivizing the solving of key technical challenges.

First, open archives of papers that the government maintains, like PubMed, could be turned into structured data amenable to large scale metascience and claim aggregation. I know the US government already has an interest in doing this, though some key open questions remain unanswered. How do you determine the best models and systems for accurately extracting information from papers? How can you establish a robust way to allow researchers to flag errors and correct them? And finally, how do you create a legal regime in working with publishers to maximize the scope of available papers? For the latter, a university or a private philanthropy may be better positioned to make structured data publicly available under journal subscription terms or fair use.

Philanthropists or government funders could also coordinate to create or commission benchmarks that evaluate whether important problems have been solved. For example, an open benchmark for claim extraction from a range of different scientific article types would be extremely helpful. Ensuring the underlying data are accurate is vital for creating these evaluations. I’ve discussed opportunistically using various human-created datasets for this purpose, but a consistent problem is that errors in human data make it difficult for them to serve as a gold standard.

Finally, with structured data and benchmarks available, the government or private philanthropy could use them to incentivize groups to develop machine learning systems that meet these benchmarks. Prizes are one potentially valuable tool for this. For example, you could establish a prize for a system that accurately updates a living meta-analysis for a small set of claims. Prizes are particularly useful as signals of problem importance, and can help create vibrant ecosystems of public and private research—see, for example the role the government played in kickstarting the current work on nuclear fusion.

Conclusion

The drawbacks of the current scientific publishing system are known. Scientists agree, metascientists agree, philanthropists agree: the published PDF plus citation graph isn’t the right substrate for maintaining a representation of the evidence base in science.

The pieces needed to build the alternative either already exist or are rapidly taking shape. The community is forming around exactly this problem, with concrete partnerships and shared infrastructure.

A pilot should start on behavioral science RCTs because that’s the slice of empirical science most amenable to legibility, where replication ground truth is richest, and where the failure modes are best documented.

What’s been missing is the galvanizing mission to assemble these pieces into something that works. That’s what I’m proposing to build.

***

Back in 2024, I published a policy brief with the Institute for Progress arguing that the FDA has been wrong for treating these letters as completely confidential—in fact, while the letters might refer to occasional proprietary information from the company, the rest of these letters is immensely valuable information about how the FDA is making decisions.

Thus, these letters should presumptively be public (with anything truly confidential being redacted). I sent the policy brief to someone I first met over 10 years ago (Marty Makary, then the incoming FDA Commissioner), who seemed appreciative.

Lo and behold, in September 2025, the FDA started releasing these letters publicly. And a lot of companies weren’t too happy.

As discussed in a great piece by my friend Adam Kroetsch, a company hired a top DC law firm (Covington & Burling) to file a petition with FDA making the case that releasing these letters is illegal. Adam’s piece is worth reading, and I just wanted to add one point:

Covington’s petition lists several examples of the FDA releasing information that was supposedly “confidential.” All of the examples make no sense.

OK, that is pretty harsh for the normally serene and academic Good Science Project newsletter.

Let’s take one example.

Covington’s memo (page 6) points to a complete response letter issued to Stealth BioTherapeutics for a drug called elamipretide, which is for a very rare genetic disorder called Barth Syndrome. The FDA initially denied approval for this drug, along with a 10-page CRL that is now public, although it soon thereafter granted accelerated approval.

Why does anyone think the letter to Stealth was somehow leaking confidential information to the public?

Here are the only examples that Covington lists about confidential information in Stealth’s CRL:

For example, FDA emphasized the p values seen in the sponsor’s randomized, placebo-controlled study and raised concerns related to study endpoints. FDA cited “significant limitations” with respect to the sponsor’s externally controlled study, including “concerns with selection bias and comparability of the control to the treated arm” and “issues with the propensity score methodology.”

I’m a former lawyer myself, so I have some sympathy for the folks trying their best to make a case here, but as a PhD-researcher who works on public policy, it seems quite odd to suggest that a company’s confidential information has been released if the FDA is worried about p-values and about the validity of a propensity-score study to eliminate selection bias.

Not only are those issues are core to several decades of scholarly discussion about how to do rigorous studies, in this case they are core to the decision-making process of a government agency that regulates about 20% of the American economy! The entire public has a right to know that these standards are being upheld.

As for the reference to p-values, here is the FDA’s paragraph that is apparently of such concern:

I do not think it should be news to anyone that p-values of 0.97 and 0.89 are NOT signs of success in a clinical trial. Literally anyone who has ever taken basic statistics should know that. Covington is saying that one specific company has a proprietary interest in the idea that such p-values are a sign that the trial failed.

Here’s the other part of the FDA’s letter that Covington seems to single out as objectionable:

Issues like selection bias, sample size, and limited covariates are all completely standard issues to worry about in any statistical analysis of any issue whatsoever (from drugs to economics to psychology). The idea that one American company has a proprietary right to keep anyone else from knowing that the FDA cares about routine statistical issues? Odd.

I could go through the other examples, but that would be tedious. They all were about routine statistical or medical issues that have absolutely nothing to do with any company’s proprietary discovery, but are instead about issues like “does this biomarker actually work” or “is a p-value of 0.97 a sign of success.”

Covington’s memo says that these issues are still confidential because saying them publicly provides a “roadmap to competitors on how to achieve FDA approval without their needing to invest in the trial and error that is a necessary part of the innovation process.”

There is no need for every applicant to the FDA to “invest in the trial and error” of proposing bad statistical analyses and bad biomarkers.

Doing so is wildly inefficient for the whole drug discovery process! It would be far better for everyone if the FDA’s basic standards as to statistics and biomarkers were publicly known far in advance of conducting clinical trials and analyzing the data.

Part of the problem here is that the FDA often or mostly acts through specific decisions or through guidance documents, rather than through official regulations that everyone knows about in advance. Apparently that has created the impression amongst many industry participants that they have some proprietary right to keep everyone else from knowing how the FDA makes decisions.

That is not how to think about such a significant government agency. The FDA’s decision-making process should be as transparent as possible so that everyone else in the industry knows how best to design clinical trials, and what endpoints to use, and so forth. These are all standard statistical and medical issues, not anything that one company has the right to keep a secret.

One more thing: Before FDA Advisory Committees have their regular meetings, they post all of the documents to be discussed. For the example above (Stealth), the FDA had already posted over 200 pages of detailed information about the drug and its effects, etc. See here and here. No one has said why a short 10-page letter from the FDA would have more confidential or proprietary information than was already publicly available.

The FDA should reject Covington’s petition. And ironically, Covington’s own client would likely be better off in the long run by being able to see how the FDA makes decisions in so many other cases.

I’m reprinting it here, as it shows why NIH can be overly influenced by groupthink.

Thanks to Dorothy Bishop at Oxford for her comments!

***

At NIH, there’s one institute solely focused on mental health: NIMH, the National Institute of Mental Health. How has that research institute done at addressing mental illness in the US?

Not very well.

That’s according to Tom Insel himself, a long-time director of NIMH from 2002 through 2015.

As Insel told Wired magazine in 2017:

“I spent 13 years at NIMH really pushing on the neuroscience and genetics of mental disorders, and when I look back on that I realize that while I think I succeeded at getting lots of really cool papers published by cool scientists at fairly large costs—I think $20 billion—I don’t think we moved the needle in reducing suicide, reducing hospitalizations, improving recovery for the tens of millions of people who have mental illness.”[1]

Why would this be the case? NIMH was arguably too obsessed with one particular type of study, to the exclusion of others. That is, under Insel, NIMH turned its focus to studies that involved genetics or brain imaging, while downplaying studies on other factors (behavioral, societal, programs and interventions, etc.).

As the New York Times put it, “Government agencies, like the National Institute on Drug Abuse and the National Institute of Mental Health, continue to double down, sinking enormous sums of taxpayer money into biological research aimed at someday finding a neural signature or ‘blood test’ for psychiatric diagnoses that could be, maybe, one day in the future, useful — all while people are in crisis now.”[2]

The Times quotes a book by Insel: “The scientific progress in our field was stunning, but while we studied the risk factors for suicide, the death rate had climbed 33 percent. While we identified the neuroanatomy of addiction, overdose deaths had increased by threefold. While we mapped the genes for schizophrenia, people with this disease were still chronically unemployed and dying 20 years early.”

As Josh Dubnau, a neurobiologist at Stony Brook University, has said, “Many of us have been crying foul for years.”[3] Or as Eric Turkheimer, one of the most prominent behavioral geneticists in the US, has said, “I find Insel’s late career revelation that neurogenomics may not be the answer to mental illness profoundly infuriating. How many dollars were wasted while behavioral models were ignored at NIMH? Careers? Patient lives?”[4]

As Brad Wyble of Pennsylvania State noted, “the problem is that we don’t know what was lost, or how good behavioral work could have complemented neuro & genetics.”[5] And as John Krakauer, professor of neurology and neuroscience at Johns Hopkins said, “the neglect of behavioral research is a source of great shame. This belief in genes and magic bullets is a cult.”[6]

In other words, the NIH spent some $20 billion on 13 years of research that didn’t make any actual progress against our nation’s enormous mental health problem. And the person in charge may be largely to blame here, by his own account!

Thus, we have a story of how one NIH Institute Director can create groupthink in an entire field. The lesson is that NIH should have tried to create structural alternatives so that one person’s view (however enlightened and wise it might have seemed at the time) can’t dominate an entire field of research.

Mental Health in America

Mental health is both a monumental societal problem and a monumental scientific mystery. It’s one of the most important subjects that an agency like NIH could address.

As for the societal problem, many entire books have been written about this, so I’ll have to limit myself to a few datapoints about the national picture.

For an initial data point, suicide is one of the leading causes of death in the United States. The CDC reports that there were 45,979 suicides in 2020. But that just scratches the surface: that same year, “an estimated 12.2 million American adults seriously thought about suicide, 3.2 million planned a suicide attempt, and 1.2 million attempted suicide.”[7]

Just going by suicidal tendencies alone, the U.S. isn’t doing so well. But mental health can go wrong in many other ways. As of 2020, around 21 million US adults had major depression,[8] while over 10 percent of children are diagnosed with ADHD.[9]

The most recent news continues to be more and more disturbing. In early 2023, the CDC released a report that got headlines and attention everywhere, i.e., the most recent results from a survey titled “Young Risk Behavior Survey Data Summary & Trends Report: 2011-2021.”[10] That report found that more than 40% of high school students feel “so sad or hopeless that they could not engage in their regular activities for at least two weeks during the previous year.” Alarmingly, it found that 57% of female students (up from 36% in 2011) had “persistent feelings of sadness or hopelessness.”[11]

If over half of teen girls have “persistent feelings of sadness or hopelessness,” that seems like a major societal crisis for the next generation. Indeed, just one chart[12]—about time spent with friends—captures a dramatic difference between today’s teens and teenagers in prior generations:

It seems obvious that many mental health issues won’t be solved just with more brain scans and genetic screenings. Leaving aside schizophrenia and similar disorders for the moment, the recent increase in depression and anxiety may well be a reflection of the fact that people have actual, legitimate reasons to be unhappy about their lives and their social arrangements.

Wherever that’s the case, brain scans and drugs would just be a band-aid trying to cover up the real problem.

***

To be sure, other disorders may be relatively rare, but can have dramatic ramifications for the individuals, their families, and the rest of society. For example, people with schizophrenia make up a huge component of the homeless population. It’s impossible to come up with nationally representative statistics (for obvious reasons), but over two dozen studies of homeless people estimate the rate of psychosis or schizophrenia as 21% and 10% respectively,[13] which at a minimum is probably 20 times as high as the rate in the general population.[14]

The criminal justice system unsurprisingly deals with an enormous number of folks who are struggling with mental health issues. By the most recent official estimate, about 43% of state prisoners “had a history of a mental health problem.”[15] Despite all of these serious mental health issues, we have been on a national crusade to shut down mental health institutions for the past several decades. As of 2020, there were only 14,187 mental health facilities of all kinds in the US.[16] Almost all of them are either outpatient or community health centers.

As a result, we don’t have the institutional capacity to deal with people who have severe mental health problems. Instead, they end up on the street and in jail.

Indeed, there’s a grim joke in the criminal justice community:

Q. “What’s the largest mental hospital in the US?”

A. “The Los Angeles County jail.”

Obviously, the Los Angeles County jail isn’t an actual mental hospital with personnel, policies, and practices created to address mental health issues. But that’s the point. It’s a default mental hospital, because it ends up handling (however inadequately) many people with severe mental illnesses who act out and have nowhere else to go. As one expert told NPR, “Many times individuals who really do require intensive psychiatric care find themselves homeless or more and more in prison. Much of our mental health care now for individuals with serious mental illness has been shifted to correctional facilities.”[17]

In the past, mental hospitals were often far from adequate. There were good reasons that people in the mid-20^th century thought that mental institutions should be abolished or cut significantly. But in abolishing many mental hospitals, we ended up putting the same people in jail instead. However bad mental hospitals used to be, jails are probably worse.

In a short-sighted attempt to do good, we made people’s lives unequivocally worse. We need a better path for providing treatment to people who are, at least for some time, largely unable to function in society.

***

As for the scientific mystery of mental health, this New York Times quote is on point:

“There have been only two major drug discoveries in the field in the past century; lithium for the treatment of bipolar disorder in 1949 and Thorazine for the treatment of psychosis in 1950. Both discoveries were utter strokes of luck, and almost every major psychiatric drug introduced since has resulted from small changes to Thorazine. Scientists still do not know why any of these drugs actually work.”[18]

In a way, it isn’t surprising that we don’t know very much about mental health and psychiatry. The human brain is unfathomably complex: Some 86 billion neurons with 1,000 to 10,000 or more connections each, plus 36-39 billion glial cells, receiving inputs from 12 million olfactory receptor cells and 140 million retinal receptor cells (among other things). The length of the nerve fibers in your brain, if stretched end to end, would be enough to stretch around the Earth four times![19]

Put all this together, and the real question might be: How does that ever work at all? How do any of us function with some semblance of normality?

There are so many ways that the brain can go wrong, and so few of those ways that we even remotely understand, that mental health is probably one of the hardest scientific problems in existence.

***

Indeed, many experts have argued that we don’t really have a solid understanding of basic issues like: 1) which mental disorders actually exist, 2) how to objectively diagnose them, 3) when and how multiple disorders occur simultaneously, and the like.

Not that there isn’t a ton of scholarship relating to these questions, to be sure! But a traditional tool of psychiatry—the Diagnostic and Statistical Manual of Mental Disorders (or DSM)—is widely acknowledged as deeply flawed in the way it draws up lists of (mostly self-reported) symptoms as to many different possible disorders.

For example, the current DSM lists many different types of anxiety-related disorders, along with diagnostic criteria, as if they are all separate conditions: “Separation anxiety disorder, selective mutism, specific phobia, social anxiety disorder, panic disorder, panic attack specifier, agoraphobia, generalized anxiety disorder, substance/medication-induced anxiety disorder, anxiety disorder due to another medical condition, other specified anxiety disorder, and unspecified anxiety disorder.”[20]

But one recent study found that mental disorders aren’t so easily demarcated: in a database of nearly 6 million medical records from Denmark, “every single mental disorder predisposed the patient to every other mental disorder—no matter how distinct the symptoms.”[21] Due to this and other research, a news article in Nature concluded that “the idea that mental illness can be classified into distinct, discrete categories such as ‘anxiety’ or ‘psychosis’ has been disproved to a large extent.”[22]

In other words, it may not make sense to try to classify all possible mental disorders (let alone 12 different types of anxiety) by a list of their symptoms, as if each is a distinct and stable disease. As Steve Hyman of Harvard (a former director of NIMH who actually helped draft the DSM) told Nature, “Any clinician could have told you that patients had not read the DSM and didn’t conform to the DSM.”

And as one scholar put it, an even more fundamental problem is that “to attempt to diagnose illness using patient symptoms resembles the approach of the eighteenth, not twenty-first-century medicine.”[23] For example, cardiologists have better ways of distinguishing between mild heart attacks and indigestion without being stuck with the observation that “the patient complains of chest pain.”

It should therefore be little surprise that when one set of scholars reviewed 102 meta-analyses covering 3,782 RCTs in mental health, they found that the risk of bias was high, and that there were “small benefits overall” for both psychotherapy and pharmaceuticals,. They concluded that “improving treatment strategies for mental disorders can be regarded as a central health challenge of the 21^st century.”[24]

A final introductory note: drug development for mental health is extraordinarily hard.

Dr. William Potter, who was once a top researcher at the mental health institute and retired last year as the vice president of translational neuroscience at the giant drug maker Merck, said that far more basic research needed to be done on the causes of mental illness before anyone -- industry or government -- could successfully create breakthrough drugs. “We still don’t even understand how lithium works,” Dr. Potter said. “So how do people think we can find drugs systematically for mental illness?”[25]

With that as the depressing backdrop, here’s a short overview of the history of US government funding for mental health research:

A Brief History of NIMH

The National Institute of Mental Health is one of the oldest NIH Institutes—it was officially launched in 1949, as called for by the National Mental Health Act of 1946 (signed by President Truman).[26] Due to the randomness of politics, NIMH has taken many forms over the decades. For example, in 1967, it was split off from NIH, and as of 1968, it became part of a larger agency that no longer exists (the Health Services and Mental Health Administration).[27] In 1972, the National Institute on Drug Abuse was launched as part of NIMH, and in 1973, NIMH “temporarily rejoined NIH,” but then yet another agency was created (the Alcohol, Drug Abuse, and Mental Health Administration) that would be composed of NIMH, NIDA, and the National Institute on Alcohol Abuse and Alcoholism. That lasted until 1989, when Congress abolished that overarching agency, and as of 1992, it gave the research components of NIMH, NIDA, and NIAAA back to NIH, although the treatment programs were split off into yet another agency (Substance Abuse and Mental Health Services Administration).

Whew! And that wasn’t even all of it.
Indeed, the tortuous history of NIMH might help explain why this book doesn’t usually delve into all the administrative history of the rest of NIH. It can all be a bit mind-boggling, and only serves to prove that politicians and experts need several swings at the bat in order to get it right (if they ever do).

Tom Insel Arrives on the Scene

In 2002, Tom Insel was named director of NIMH. As a bit of background, he got a medical degree from Boston University, and then did clinical training in psychiatry at UCSF from 1976 to 1979. From there, he went to work at NIMH as a clinical fellow. After many years there, he went to Emory University in Atlanta in 1994 to direct a research center on primates, and in 1999, he began leading a major NSF-sponsored Center for Behavioral Neuroscience.[28] That was his final appointment before becoming NIMH Director in 2002.

Insel ended up taking NIMH in a very particular direction as to what was and wasn’t fundable. He preferred to fund studies involving neuroimaging and/or genetics, in an attempt to uncover the actual biological mechanisms in the brain that may underlie various mental illnesses.

Indeed, in 2010-11, NIMH announced (with some justification) that it was unhappy with the DSM and the traditional ways of diagnosing mental illness. As Steve Hyman (himself a former NIMH director) told Science, the current DSM model of diagnosing mental illness made little sense. For example, a diagnosis of major depression could be made if a patient had at least five of nine symptoms. But “in this scenario, it’s possible for two patients to receive the same diagnosis with only one symptom in common.”[29] Such a system of diagnosing disease struck many people as less than objective or satisfactory.

NIMH therefore launched an approach called Research Domain Criteria (RDoC) as a replacement for the traditional way of classifying and diagnosing mental disorders. As an NIMH researcher told Science in 2010, “What we are doing is trying to develop new ways to classify disorders that are based on identifiable neural circuits.”[30] The intent was to divide up psychiatry into five broad domains that are “present in everyone but whose extremes correspond to mental illness: negative emotionality, positive emotionality, cognitive processes, social processes, and arousal/regulatory systems.”[31] Then, in 2012, Insel co-authored a scholarly article arguing that psychiatric diagnoses had no “biological ‘gold standard’ definition,” that we are left with “a profusion of statistically significant, but minimally differentiating, biological findings,” and that rather than sticking with the DSM-listed disorders, we should focus on developing new ways to identify the underlying biological differences.[32]

By 2013, Insel drew attention—not all of it positive—by writing a blog post formally announcing that NIMH’s funding would no longer look to the American Psychiatric Association’s famous DSM.[33] In his words, the “weakness” of DSM is its “lack of validity.” Unlike the way we define heart disease, cancer, or AIDS, “the DSM diagnoses are based on a consensus about clusters of clinical symptoms, not any objective laboratory measure. In the rest of medicine, this would be equivalent to creating diagnostic systems based on the nature of chest pain or the quality of fever.” He then announced that NIH’s RDoC project (“Research Domain Criteria”) would “transform diagnosis” by including “genetics, imaging, cognitive science,” and more.

As the New York Times reported, “Just weeks before the long-awaited publication of a new edition of the so-called bible of mental disorders, the federal government’s most prominent psychiatric expert has said the book suffers from a scientific ‘lack of validity.’”[34] The Times quoted Insel as saying: “As long as the research community takes the D.S.M. to be a bible, we’ll never make progress. People think that everything has to match D.S.M. criteria, but you know what? Biology never read that book.”

At the same time, the Times itself was well aware of the many difficulties at hand:

Decades of spending on neuroscience have taught scientists mostly what they do not know, undermining some of their most elemental assumptions. Genetic glitches that appear to increase the risk of schizophrenia in one person may predispose others to autism-like symptoms, or bipolar disorder. The mechanisms of the field’s most commonly used drugs — antidepressants like Prozac, and antipsychosis medications like Zyprexa — have revealed nothing about the causes of those disorders. And major drugmakers have scaled back psychiatric drug development, having virtually no new biological “targets” to shoot for.[35]

Insel’s blog post was widely criticized, and not just by psychiatrists who thought their main body of work had been insulted. Dorothy Bishop at Oxford, for example, saw “big problems” with the new NIMH approach.[36] For example, the NIMH’s approach failed to include “anything about experience or environment,” and instead assumed that all mental problems are “disorders of brain circuits.” Relatedly, she thought there was no evidence that “better knowledge of neurobiological correlates” would help improve psychological interventions for, say, obsessive-compulsive disorder. For another example, the NIMH proposal took a “naïve view of the potential of genetics” to improve psychiatry. Most behaviors have, at most, a very tiny association with any gene, and pouring money into genetics at present “sounds to me like a recipe for wasting a huge amount of research funding.”

Bishop was far from alone in her skepticism about NIMH’s turn towards genetics and imaging. As Nature pointed out, “clinical researchers bristled in 2013 when former NIMH director Thomas Insel announced that the agency would shift away from funding research that classified people using DSM-5, and again in 2014 when Insel said that the NIMH would not fund clinical trials that didn’t seek to understand the biological mechanism underlying a particular treatment or illness.”[37]

The New York Times wrote in 2014 that Insel’s tenure at NIMH created a “departure” that was “far larger than just about anyone could have anticipated.” He “sharply shifted the agency’s focus—to basic neuroscience and genetics, at the expense of the very type of behavioral research he himself had once done.”[38] He became convinced that “the only way to build a real psychiatric science is from first principles — from genes and brain biology, as opposed to identifying symptom clusters. Some of the mental health institute’s largest outlays under Dr. Insel have been to support projects that, biologically speaking, are like mapping the ocean floor.”[39] As one respected researcher from Duke told the Times, “N.I.M.H. is betting the house on the long shot that neuroscience will come up with answers to help people with serious mental illness. . . . It does little or no psychosocial or health services research that might relieve the current suffering of patients.”[40]

How did NIMH’s priorities arguably distort research? One notable example: Kristina Olson was a psychology professor at Princeton who studied gender and sex issues. She wrote that she and her colleagues “were so desperate for funding for our trans work” that they considered flying patients “to Seattle to put them in a scanner, [and] then get out to do the actual study we thought mattered (but wasn’t getting funded). That is how crazy this is. It would have added a million $ & been more fundable.”[41]

In other words, she couldn’t get her actual study funded, but thinks that if she had added a million dollars in funding so as to fly people around the country to have their brains scanned, that would have made NIMH under Tom Insel more likely to fund her work.

Here’s another example of how NIMH’s approach was arguably too narrow. NIMH issued a funding announcement on eating disorders in 2013. It stated that the primary goals were to:

“(1) support integrative, hypothesis-driven studies of neural circuits and/or other biological mechanisms underlying eating disorders; (2) support the use of dimensional constructs (defined for the purposes of this FOA below) as a primary means to investigate these mechanisms; (3) support the delineation of trajectories over time (e.g., across developmental stages or across illness course); (4) encourage integration across different levels of analysis (e.g., behavior, cells, circuits, genes, molecules, physiology, self-report, symptoms); (5) encourage neurodevelopmental research in eating disorders; and (6) encourage application of systems neuroscience methods to the study of eating disorders.”[42]

That’s fairly dense. What does it all mean?

As psychology professor Sanjay Srivastava wrote at the time,[43] those funding priorities were all about neuroscience, genes, biological mechanisms, etc. But if you actually want to address eating disorders, some important words and concepts were entirely missing: “social; media; culture; family; peer (when not followed by ‘review’ referring to the funding processes); body image; self.”

As he put it, “if NIMH thinks that basic research on media, on family environments, on peer influence, on self-concept, on cultural norms are not terribly important for understanding and treating eating disorders — well, that’s really hard to defend.”

As one might suspect, one fairly dramatic change at NIMH over these years was that it stopped supporting as many clinical trials on mental disorders. One analysis by Nature in 2017 found that “the number of clinical trials funded by the National Institute of Mental Health has fallen by 45% since the agency began to focus on the biological roots of disease.”[44] As that article pointed out, “The NIMH’s embrace of fundamental research has infuriated many clinical researchers, who see it as an attempt to invalidate their methods — and say that there is scant evidence to support the idea that using RDoC will lead to greater insight or better treatments for mental illness.” As Dr. Allen Frances from Duke told the New York Times, “Instead of being an institute of mental health, he has made it almost exclusively a brain research institute.”[45] Another review article found that “between 2006 and 2023, NIMH-funded extramural drug trials for schizophrenia and bipolar disorder decreased by 95%”![46]

As well, NIMH seemed to lean into the trend of launching well-funded initiatives or moonshots to much fanfare and publicity. The results were often impressive, but often criticized as well, and it’s not clear how much they have yet contributed to improving mental health in the real world.

One major effort from NIMH at the time was the Psychiatric Genomics Consortium.[47] The claim was that “by bringing together genetic data from hundreds of thousands of individuals around the world, the PGC is working to find the genetic variants that change risk of disease. Identifying the genetic basis of these disorders helps us to understand the underlying biology and to develop better ways of preventing and treating disorders.”[48] This consortium seems to be widely accepted as a success, at least as to genetic data, but as of yet there is no evidence that they have improved anything for patients.

Another major effort – the Human Connectome Project – was “launched in 2009” as an “ambitious effort to map the neural pathways that underlie human brain function.”[49] It officially got to work in 2010 when it “awarded $40 million to two collaborating research consortia to map the human brain’s connections in high resolution.” The first consortium (Minnesota, Oxford, and Washington University in St. Louis) “set out to comprehensively map human brain circuitry in 1200 healthy adults using cutting-edge methods of noninvasive neuroimaging.” The other consortium (UCLA, MGH/Harvard) tried to “create a new magnetic resonance imager optimized for measuring connectome data.”[50]

The results? A 2021 article by the project leaders noted that “more than 27 Petabytes of data have been shared, and 1,508 papers acknowledging HCP data use have been published.”[51] The article also noted “several scientific advances,” including “improved cortical parcellations, analyses of connectivity based on functional and diffusion MRI, and analyses of brain-behavior relationships.” Even accepting all of these papers at face value, I don’t see much here that actually improves the situation for US mental health, and while I reached out to study leaders for comment, they did not respond with any explanation as to how all of this MRI-type research could, even in theory, directly solve mental health problems.

Then, in 2015, NIMH and a few other Institutes and Centers joined to fund the so-called “ABCD Consortium,” which would scan the brains of over 10,000 children, and follow them into adulthood. At a cost of $300 million, the study will track quite a number of things, including “how childhood experiences (such as sports, videogames, social media, unhealthy sleep patterns, and smoking) interact with each other and with a child’s changing biology to affect brain development and social, behavioral, academic, health, and other outcomes.”[52] A New York Times article dismissively said that there are “so many interacting variables of experience and development that it’s hard to discern what the study’s primary goals are.”[53]

As well, the NIH has a $50 million project called the PsychENCODE project that is a “collaboration between 15 research institutes working to provide an enhanced framework of regulatory genomic elements in individuals with neuropsychiatric disorders.”[54] What does this mean? In large part, scanning over 2,000 brains to look for genetic links to things like schizophrenia, autism, and bipolar disorder. A critic of the project (Dr. Dan Graur, an evolutionary geneticist in Texas) said, “If you take something vague [i.e., a diagnosis of schizophrenia] and correlate it with millions of genetic and epigenetic variations, you are bound to get statistical significance that will have little biological significance.”[55] As the Times noted, the prominent psychological researcher Scott Lilienfeld said of such projects, “They’re either fishing expeditions or Hail Marys. Take your pick.”[56]

There is good reason to be skeptical that neuroimaging is anywhere near being able to help with clinical approaches. One paper in Nature recently found that brain-wide association studies (where the researchers try to associate cognitive or psychiatric characteristics with brain imaging data) are typically far too small, with a median sample size of 25 even though you would need thousands of participants to get accurate results.[57]

To make matters worse, another recent paper found that current machine learning approaches can’t diagnose major depressive disorder from neuroimaging data. Specifically, the study looked at 1,801 patients with major depression, and compared them to healthy patients based on detailed neuroimaging data.

The results: Even with 2.4 million machine learning models, the accuracy of predicting a depression diagnosis never rose about 62%, not much better than flipping a coin. The conclusion: “although multivariate neuroimaging markers increase predictive performance compared to univariate analyses, classification on the level of the individual patient—even under optimal conditions—does not reach clinically relevant levels.”[58]

Keep in mind that this study was larger and more sophisticated than most brain imaging studies—as Dorothy Bishop of Oxford said to me, maybe we should not be funding those smaller studies in the first place.

The story of another paper is remarkable. In 2017, Nature Human Behaviour published what seemed to be a blockbuster paper (sponsored in part by an NIMH grant).[59] The paper claimed that when you put suicidal youth into an fMRI machine and looked at neural representations associated with concepts like “death,” “cruelty,” “trouble,” “carefree,” “good,” and “praise,” a machine learning algorithm was able to distinguish them from healthy control subjects with 91% accuracy, and that an algorithm could even tell with 94% accuracy who had actually tried to commit suicide versus who had not. As neuroscientist Konrad Kording said, “If true, the paper’s approach could revolutionize psychiatric approaches to suicide.”[60]

But it wasn’t true, as Kording himself and a colleague showed. A full six years later in 2023—after several years of effort—the paper was retracted. The paper—which had only 17 suicidal subjects and 17 controls—was full of overfitting. A rigorous way to do machine learning on medical data (or any other data, for that matter) is to divide up the dataset ahead of time, and to reserve some of the data (called a “test set”) to be used only at the very end of the study, as a final test of whether the algorithm you created on the other data is usable on data that is still fresh, so to speak. But here, as Kording said, “The authors apparently used the test data to select features. Obvious mistake.”[61]

The result: Overfitting. It’s a little bit analogous to shooting an arrow into a wall, and then painting a target around where the arrow landed. You can’t use where the arrow landed to pick where the target should be. That’s backward. The target has to be picked ahead of time, and then tested fairly.

The problem with neuroscience may go much deeper, however. As neuroscientist Erik Hoel points out in his book The World Behind the World, we don’t even know that average brain activity (which is behind “almost all the reported effects in neuroscience”) is a meaningful thing to measure and interpret. He points to animal studies finding that an individual animal’s brain activity and behavior didn’t necessarily correspond either with average brain activity or average behavior, indicating “that the statistical constructs used to create neuroscience papers are epiphenomena for the brain itself—as if we are trying to understand a clock by the shadows it casts.”[62]

Another classic article in neuroscience showed that if you analyze a microprocessor by limiting yourself to the typical neuroscience techniques, you’d never come close to understanding the inner workings of the microprocessor. This “suggests that the availability of unlimited data, as we have for the processor, is in no way sufficient to allow a real understanding of the brain.”[63]

In other words, while we should fund some neuroimaging and machine learning studies, we shouldn’t put all our eggs in that basket while millions of people are struggling with mental health.

The same may be true of genetics. Despite the amazing advances in that field in the past 20-25 years, one recent article pointed out that “it is still not possible to cite a single neuroscience or genetic finding that has been of use to the practicing psychiatrist in managing these illnesses despite attempts to suggest the contrary.”[64] “Rather than neuroscience research, serendipity or lateral thinking remain the key tools in psychotropic drug discovery.” The article goes on to make the common sense point that we ought to diversify our mental health funding: “Investment in psychological, social science and service delivery research has occurred but represents only a small fraction of the attention and funding of neuroscience. . . . Is it not time for research to refocus resource and expertise away from the laboratory and onto these more relevant psychological and social sciences and research into clinical practice, public health and service delivery?”

You might think that the scholar here might have overstated things in saying that nothing from neuroscience or genetics has proven useful to the practicing psychiatrist. And a commentary on that article does make that counterargument: “The most recent genome-wide association study on depression found 87 independent loci that were associated with depression, with a startling lack of genes involved in the 5-HT system. . . . Findings such as these are likely to be of great benefit in developing new treatments.”[65]

Maybe so. But if you follow the reference for that claim, you end up with an article that performed a heroic amount of work—analyzing data on 807,553 individuals from the largest genome-wide association studies on depression, along with an ”independent replication sample of 1,306,354 individuals,” in order to narrow in on 87 genetic variants, all of which put together explained only 0.8% to 3.2% of the variance in depression.[66]

In other words, the scholars studied thousands of genes across over 2 million patients, and in the end were only able to find 87 genes that collectively explain at most about 3% of why someone gets depressed (and nothing about the remedy, because this study didn’t even look at that). While an impressive achievement, this sort of study is still nowhere near being able to give specific clinical advice on how psychiatrists should treat an actual patient with depression.

In short, we need mental health funding that isn’t consumed with the latest fad (genetics, machine learning, the microbiome, or whatever). Mental health is an incredibly complex problem that needs attention at many different levels—yes, genetics and machine learning, but also community-based interventions, and therapeutic programs, and more. The most complex problem in the universe doesn’t need the most simplistic approach.

Taking Stock: How Is Mental Health Funding Today?

Even now, several years after Insel’s departure from NIMH, researchers are still feeling the impact. One neuroscientist from Princeton recently wrote that funding priorities at NIMH are the “exact opposite” from what we need to actually understand mental health:

“As examples, a program officer at the National Institute for Mental Health (NIMH) requested that a colleague withdraw her funding application from consideration prior to review, saying that regardless of reviewers’ evaluation, the clearly mental-health relevant research will not be funded as it does not include a neural component; I have similarly been told, on consultation with several program officers, that a computational psychiatry center that focuses on behavioral measures is not of interest to NIMH unless we include neuroscience methods such as fMRI or MEG, despite research so far showing little return for such techniques in understanding mental illness (Roiser, 2015), not to mention in developing clinically-feasible tools for diagnosis and treatment selection.”[67]

As a number of prominent psychiatrists who had served on NIMH’s Advisory Council wrote in 2016, “diversification is a prudent strategy” in any investment portfolio, and a “disproportionate investment in neuroscience is as imprudent as investing only in growth stocks and neglecting less risky investments that yield immediate albeit potentially more modest benefits.”[68] And yet, at the time, basic and translational neuroscience research made up some 85% of NIMH’s funding, with only 15% going towards services, programs, and interventions.

From a structural perspective, it seems like a bad idea to give an NIH Institute Director the power to make radical changes in any one direction, turning the entirety of a given scientific field into his or her personal plaything.

Sure, a Feynman or Einstein or Curie might be able to wield that power wisely, but the vast majority of people won’t. By far, most scientists (especially the type of person who often rises to the top of a government bureaucracy) should be more prudent and humble about whether their own personal views should dominate an entire field.

NIH should give more thought towards preventing what happened at NIMH under Tom Insel. Perhaps NIH should engage in “red teaming” exercises (a term from military and intelligence sources, where a team of people is deliberately tasked with trying to poke holes and critique the current approach).

As well, NIH should systematically set aside 20% or more of its funding to ideas, theories, and lines of research that aren’t currently in favor. At the other end of the spectrum, NIH should set aside 20% of funding for replication studies and methods development/evaluation so that we know what is actually reliable in this area.

Nothing in the history of science would make us think that it’s a good idea for one person or one idea to gain such prominence that they are able to squelch other approaches. And as shown in other chapters, the structure and model of NIH (with a fairly rigid system of peer review) is biased towards creating groupthink, rather than combatting it.

[1] Adam Rogers, “Star Neuroscientist Tom Insel Leaves the Goodle-Spawned Verily for . . . a Startup?,” Wired (May 11, 2017), available at https://www.wired.com/2017/05/star-neuroscientist-tom-insel-leaves-google-spawned-verily-startup/.

[2] Benedict Carey, “Science Plays the Long Game. But People Have Mental Health Issues Now,” New York Times (April 1, 2021), available at https://www.nytimes.com/2021/04/01/health/mental-health-treatments.html.

[3] See his tweet of Feb. 22, 2022, at

.

[4] See his tweet of Feb. 22, 2022, at

https://twitter.com/bradpwyble/status/1496334071315714048

.

[5] See his tweet of Feb. 22, 2022, at

https://twitter.com/bradpwyble/status/1496334071315714048

.

[6] See his tweet of Feb. 22, 2022, at

.

[7] See https://www.cdc.gov/suicide/facts/index.html.

[8] See https://www.nimh.nih.gov/health/statistics/major-depression.

[9] See https://www.nimh.nih.gov/health/statistics/attention-deficit-hyperactivity-disorder-adhd.

[10] See https://www.cdc.gov/healthyyouth/data/yrbs/pdf/YRBS_Data-Summary-Trends_Report2023_508.pdf.

[11] https://www.cdc.gov/healthyyouth/data/yrbs/pdf/YRBS_Data-Summary-Trends_Report2023_508.pdf#page=64.

[12] See

After Babel

The new CDC report shows that Covid added little to teen mental health trends

For the first few months of this new substack, I plan to publish a major post every two or three weeks. In the weeks in between, like today, I’ll sometimes write something shorter and respond to the best criticisms of my previous post. I’ll usually write these responses with Zach Rausch, the lead researcher for this substack…

Read more

3 years ago · 169 likes · 87 comments · Jon Haidt

.

[13] Getinet Ayano, Getachew Tesfaw, and Shegaye Shumet, “The prevalence of schizophrenia and other psychotic disorders among homeless people: a systematic review and meta-analysis,” BMC Psychiatry 19 (2019), available at https://bmcpsychiatry.biomedcentral.com/articles/10.1186/s12888-019-2361-7.

[14] See https://www.nimh.nih.gov/health/statistics/schizophrenia.

[15] Bureau of Justice Statistics, Indicators of Mental Health Problems Reported by Prisoners: Survey of Prison Inmates, 2016 (published in 2021), available at https://bjs.ojp.gov/library/publications/indicators-mental-health-problems-reported-prisoners-survey-prison-inmates.

[16] See the National Mental Health Services Survey: 2020, available at https://www.samhsa.gov/data/sites/default/files/reports/rpt35336/2020_NMHSS_finsal.pdf, p. 9, Table 1.1.

[17] Samantha Raphelson, “How The Loss Of U.S. Psychiatric Hospitals Led To A Mental Health Crisis,” NPR (Nov. 30, 2017), available at https://www.npr.org/2017/11/30/567477160/how-the-loss-of-u-s-psychiatric-hospitals-led-to-a-mental-health-crisis.

[18] See https://www.nytimes.com/2011/01/23/health/policy/23drug.html?pagewanted=all.

[19] See Bente Pakkenberg et al., “Aging and the human neocortex,” Experimental Gerontology 38 (2003): 95-99.

[20] See https://dsm.psychiatryonline.org/doi/full/10.1176/appi.books.9780890425787.x05_Anxiety_Disorders.

[21] Michael Marshall, “The hidden links between mental disorders,” Nature (May 5, 2020), available at https://www.nature.com/articles/d41586-020-00922-8.

[22] Michael Marshall, “The hidden links between mental disorders,” Nature (May 5, 2020), available at https://www.nature.com/articles/d41586-020-00922-8.

[23] Andrew Scull, “American psychiatry in the new millennium: a critical appraisal,” Psychological Medicine (2021): 1-9.

[24] Falk Leichsenring et al., “The efficacy of psychotherapies and pharmacotherapies for mental disorders in adults: an umbrella review and meta-analytic evaluation of recent meta-analyses,” World Psychiatry (Jan. 11, 2022), available at https://onlinelibrary.wiley.com/doi/10.1002/wps.20941.

[25] See https://www.nytimes.com/2011/01/23/health/policy/23drug.html?pagewanted=all.

[26] See https://web.archive.org/web/20070310071549/https://www.nih.gov/about/almanac/archive/1999/organization/nimh/history.html. See also https://govtrackus.s3.amazonaws.com/legislink/pdf/stat/60/STATUTE-60-Pg421.pdf.

[27] See https://web.archive.org/web/20070310071549/https://www.nih.gov/about/almanac/archive/1999/organization/nimh/history.html.

[28] See https://en.wikipedia.org/wiki/Thomas_R._Insel.

[29] Greg Miller, “Beyond DSM: Seeking a Brain-Based Classification of Mental Illness,” Science327 no. 5972 (2010): 1437.

[30] Greg Miller, “Beyond DSM: Seeking a Brain-Based Classification of Mental Illness,” Science327 no. 5972 (2010): 1437.

[31] Greg Miller, “Beyond DSM: Seeking a Brain-Based Classification of Mental Illness,” Science327 no. 5972 (2010): 1437.

[32] S. Kapur, A. G. Phillips, & T. R. Insel, “Why has it taken so long for biological psychiatry to develop clinical tests and what to do about it?,” Molecular Psychiatry 17 (2012): 1174-79.

[33] An archived version of his blog post is here: https://web.archive.org/web/20130503094041/http://www.nimh.nih.gov/about/director/2013/transforming-diagnosis.shtml.

[34] Pam Belluck and Benedict Carey, “Psychiatry’s Guide Is Out of Touch With Science, Experts Say,” New York Times (May 6, 2013), available at https://www.nytimes.com/2013/05/07/health/psychiatrys-new-guide-falls-short-experts-say.html?pagewanted=all.

[35] Pam Belluck and Benedict Carey, “Psychiatry’s Guide Is Out of Touch With Science, Experts Say,” New York Times (May 6, 2013), available at https://www.nytimes.com/2013/05/07/health/psychiatrys-new-guide-falls-short-experts-say.html?pagewanted=all.

[36] See http://deevybee.blogspot.com/2014/05/changing-landscape-of-psychiatric.html.

[37] Sara Reardon, “US mental-health agency’s push for basic research has slashed support for clinical trials,” Nature 546, 339 (2017), available at https://www.nature.com/articles/546338a.

[38] Benedict Carey, “Blazing Trails in Brain Science,” New York Times (Feb. 3, 2014), available at https://www.nytimes.com/2014/02/04/science/blazing-trails-in-brain-science.html.

[39] Benedict Carey, “Blazing Trails in Brain Science,” New York Times (Feb. 3, 2014), available at https://www.nytimes.com/2014/02/04/science/blazing-trails-in-brain-science.html.

[40] Benedict Carey, “Blazing Trails in Brain Science,” New York Times (Feb. 3, 2014), available at https://www.nytimes.com/2014/02/04/science/blazing-trails-in-brain-science.html.

[41] See her tweet of April 2, 2021, at

https://twitter.com/olsonista/status/1377981103592574992

.

[42] See https://grants.nih.gov/grants/guide/rfa-files/RFA-MH-14-030.html.

[43] See https://thehardestscience.com/2013/05/24/where-is-rdoc-headed-a-look-at-the-eating-disorders-foa/.

[44] Sara Reardon, “US mental-health agency’s push for basic research has slashed support for clinical trials,” Nature 546, 339 (2017), available at https://www.nature.com/articles/546338a.

[45] See https://www.nytimes.com/2014/02/04/science/blazing-trails-in-brain-science.html.

[46] See E. Fuller Torrey, Wendy Simmons, and Lisa Dailey, “Schizophrenia, clinical and basic research at NIMH: A 75 Year retrospective,” Psychiatry Research 342 (Dec. 2024), available at https://www.sciencedirect.com/science/article/abs/pii/S0165178124005109?dgcid=coauthor.

[47] See

https://pgc.unc.edu/

.

[48] See https://pgc.unc.edu/for-the-public/.

[49] See https://neuroscienceblueprint.nih.gov/human-connectome/connectome-programs.

[50] See https://neuroscienceblueprint.nih.gov/human-connectome/connectome-programs.

[51] See Jennifer Stine Elam et al., “The Human Connectome Project: A Retrospective,” Neuroimage 244 (2021), available at https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9387634/.

[52] See https://abcdstudy.org/about/.

[53] Benedict Carey, “Science Plays the Long Game. But People Have Mental Health Issues Now,” New York Times (April 1, 2021), available at https://www.nytimes.com/2021/04/01/health/mental-health-treatments.html.

[54] See

http://www.psychencode.org/

.

[55] See Kelly Servick, “Genomic data from 2000 human brains could reveal roots of schizophrenia, autism, and other neurological disorders,” Science (Dec. 13, 2018), available at https://www.sciencemag.org/news/2018/12/genomic-data-2000-human-brains-could-reveal-roots-schizophrenia-autism-and-other.

[56] Benedict Carey, “Science Plays the Long Game. But People Have Mental Health Issues Now,” New York Times (April 1, 2021), available at https://www.nytimes.com/2021/04/01/health/mental-health-treatments.html.

[57] Scott Marek et al., “Reproducible brain-wide association studies require thousands of individuals,” Nature 603 no. 7902 (2022): 654-660.

[58] Nils R. Winter et al., “A Systematic Evaluation of Machine Learning-based Biomarkers for Major

Depressive Disorder across Modalities” (Feb. 2023), preprint available at https://www.medrxiv.org/content/10.1101/2023.02.27.23286311v1.

[59] Marcel Adam Just et al., “Machine learning of neural representations of suicide and emotion concepts identifies suicidal youth,” Nature Human Behaviour 1 (2017): 911-919.

[60] See

.

[61] See

.

[62] Erik Hoel, The World Behind the World: Consciousness, Free Will, and the Limits of Science (2023), pp. 54-55.

[63] Eric Jonas and Konrad Paul Kording, “Could a Neuroscientist Understand a Microprocessor?,” PLoS Computational Biology (Jan. 12, 2017), available at https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1005268.

[64] David Kingdon, “Why hasn’t neuroscience delivered for psychiatry,” BJPsych Bulletin 44 no. 3 (Feb. 13, 2020): 107-09, available at https://www.cambridge.org/core/journals/bjpsych-bulletin/article/why-hasnt-neuroscience-delivered-for-psychiatry/2EB9F2202E61BCC98A5D1E5F5F825607.

[65] Lindsey Isla Sinclair, “What neuroscience has already done for us: Commentary on ‘Why hasn’t neuroscience delivered for psychiatry?,’” BJPsych Bulletin 44 no. 3 (June 2020): 110-112, available at https://www.cambridge.org/core/journals/bjpsych-bulletin/article/what-neuroscience-has-already-done-for-us/73A1986D1F5097D6555474ECAD8F63B3.

[66] See Table 2 in David M. Howard et al., “Genome-wide meta-analysis of depression identifies 102 independent variants and highlights the importance of the prefrontal brain regions,” Nature Neuroscience 22 no. 3 (March 2019): 343-352, available at https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6522363/.

[67] Yael Niv, “The primacy of behavioral research for understanding the brain,” PsyArXiv preprint (2020), available at https://psyarxiv.com/y8mxe/. This was later published at Behavioral Neuroscience 135 no. 5 (2021): 601-609.

[68] Roberto Lewis-Fernandez, “Rethinking funding priorities in mental health research,” British Journal of Psychiatry 208 (2016): 507-09.

I originally wrote this piece a year ago, and then parts of it turned into this co-authored piece at Issues in Science and Technology. But there were several thousand words that didn’t end up being published. I thought it was worth resurrecting here.

Introduction

The National Institutes of Health budget was $49 billion in fiscal year 2023, while the National Science Foundation’s budget was nearly $10 billion. The overall goal is to improve scientific advancement. How are NIH and NSF doing with some $59 billion in public funds?

It might be surprising, but no one actually knows very much about a wide range of important questions, such as:

• Is it better to give more funding to “the person not the project”? “The team not the project”? “The institution not the project”?

• Is it better to give 8- or 10-year grants as opposed to 3-4 year grants?

• Is it better to change peer review so that a single positive vote might get something funded?

• Should we “red team” scientific fields where the funded grants mostly seem to follow one particular theory?

• Would it be better to give program officers more discretion to overrule or even bypass peer review? 

• Is it better to fund a lone assistant professor or to fund the 51^st person to join a large lab?

• What would happen if we demanded that all scientific grants publicly report at least one “failure”? Would we encourage more risk-taking and/or truth-telling?

• What types of research proposals do peer reviewers tend to reject? What is the difference between those proposals and the ones that get approved?

• How often is high-impact research turned down in initial review?

• How often do proposals below the payline nonetheless get funded? Who makes those decisions, and what is their rationale? Over time, how do those grants perform compared to grants above the payline?

• How does the NIH intramural program compare to the extramural program? 

Why don’t we know the answers to these questions?

In part, it’s because we need more experimentation with different funding models. But many of these questions could be at least partially answered by existing data that NIH and NSF usually keep hidden from scholars, despite having a slim legal basis (if any) for doing so. And even if we did more experimentation in federal science funding, external scholars would still need access to the basic underlying data in order to help out.

For example, very few scholars have ever been able to access data on successful versus unsuccessful proposals, how they fared in the peer review process, the internal reviews from agency staff and peer reviewers, etc. There are many top scholars who could make use of this data and who are frustrated by agency responsiveness (or lack thereof), but who do not want to be named in any way whatsoever for fear that NIH or NSF might somehow retaliate against them.1

***

The above is just my list of interesting metascience questions. But agencies themselves have been coming up with their own lists. The NSF’s 2022-26 Learning Agenda includes research questions such as:

• “What are the impacts of NSF policies and programs on the diversity of the STEM workforce and the participation of the most underrepresented groups?”

• “How do EPSCoR [Established Program to Stimulate Competitive Research] program funding strategies (infrastructure, co-funding, and outreach) contribute to increasing academic research competitiveness across jurisdictions?”

• “What are the benefits of receiving an award from a program supported by a partnership? How do these differ from benefits associated with awards from programs not supported by a partnership?”

• “What are the characteristics of proposals evaluated through the merit review process? Are these characteristics (of individual investigators, teams, institutions, or proposed projects) associated with different review or funding outcomes?”

• “What outcomes are associated with the adoption of a no-deadlines proposal submission process?”

Similarly, NIH posted a list of evaluation questions on the OMB Portal. Some of its questions include: 

• “How can NIH improve on identifying desired outcomes and measuring impact related to its mission?” 

• “What measures can NIH use to capture both incremental knowledge gains and failures that ultimately contribute to scientific success?” 

• “What approaches can NIH use to measure impact of different categories of science (e.g., basic, translation, clinical) and the technology and operations used to support the science?”

• “Are there methods that NIH can use to better predict and identify scientific opportunities (e.g., the emergence of gene editing technology)?”

• “Are there approaches that could inform NIH funding decisions by measuring scientific quality, rigor, and reproducibility?”

• “What evidence does NIH need to improve the clinical research ecosystem? What would inform a re-envisioning of the clinical trials system to maximize quality, participant experience, accessibility, timeliness, and impact on clinical care?”

For all of these questions and more, science funding agencies would benefit from working with external evaluators and scholars and gaining from all of their additional expertise. In fact, the agency would probably learn far more from allowing external scholars to have access to the necessary data, than from trying to analyze all of these questions in-house (where there is a serious conflict of interest).

POLICY RECOMMENDATION: If we want to know whether any government agency is well-functioning and successful, independent scholars and evaluators need access to comprehensive data on the agency’s operations and outcomes. Thus, NIH and NSF (and more) should make their internal data available to outside researchers by default given a good faith request via the Standard Application Process at ResearchDataGov, and to do so on a streamlined basis with narrow exceptions only when necessary. If agencies need extra appropriations for personnel to prepare data, Congress should step in. Moreover, agencies should not be allowed to veto publications except in the narrowest of circumstances (e.g., malfeasance).

The Evidence Act

A major theme of the Foundations for Evidence-Based Policymaking Act of 2018, more commonly known as the Evidence Act, is that federal agencies should be engaged in regular evidence-building and evaluation as to how their efforts are working.

As well, the Evidence Act included a section titled the “OPEN Government Data Act,” with OPEN standing for “Open, Public, Electronic, and Necessary.” [You might be reminded of the joke that the B in Benoit B. Mandelbrot stands for “Benoit B. Mandelbrot.”]

The point was to require federal agencies to “make data open by default” (see section 202(c) here). The statute requires all federal agencies both to “make each data asset of the agency available in an open format” and to “make each public data asset [i.e., subject to FOIA] of the agency available as an open government data asset; and under an open license.” 44 U.S.C. § 3506(b)(6). In other words, as a Data Foundation report put it, “For the first time, OPEN codifies a presumption that non-sensitive government data should be open by default.”

But agencies are sidestepping the Evidence Act as well as what other laws (such as FOIA and the Privacy Act) actually entail. Contrary to NIH and NSF’s approach, federal appellate caselaw has interpreted FOIA to actually require the release of information about unfunded proposals, and under the Evidence Act, that information should be automatically available as an open data asset.

Moreover, contrary to these agencies’ representations, the Privacy Act isn’t an automatic fallback reason to protect such information (as it has a clear exception for “statistical research”). In other words, agencies can and must do more under current law to make data available to outside researchers. Section II below discusses current law in more detail.

Another difficulty is that agencies see themselves as trapped in an outdated and binary legal regime wherein the two choices are: 1) Release data to (potentially) the entire world under FOIA, or else 2) Treat the data as protected under the Privacy Act.

We need more of an in-between legal regime for handling data that might be slightly private and that should be released to some people but not everyone. An example is peer review scores, which might be embarrassing if released to the entire world with names attached, but should most definitely be made available to independent scholars for analysis.

The last section therefore makes the case for an intermediate legal regime that would allow NIH and NSF to comply with the spirit of the Evidence Act. The lack of legal clarity here creates an environment wherein agency lawyers (who are understandably prone to be overly protective) err on the side of refusing to allow the release of data to external researchers. As a result, we are lacking as to independent analyses of how NIH, NSF, etc., are doing.

The memo concludes with several specific suggestions for how the White House could take action in order to point scientific research agencies in the right direction, as well as a bill that Congress could adopt to further clarify the agencies’ responsibilities to make data available.

Release of Research Data Under FOIA and the Privacy Act: More Is Possible or Even Required Than Agencies Admit

A.    FOIA

The Freedom of Information Act (FOIA) in 5 U.S.C. § 552 broadly requires the federal government to share information and data with the public on request. Similarly, the Government in the Sunshine Act (5 U.S.C. §552b) requires government meetings to be open to the public, with a few exceptions that are copied from FOIA.

The NIH treats FOIA as the governing law here. As the NIH Grants Policy Statement says in Section 2.3.11.2.1, if anyone other than the researcher requests data on proposals, “such requests are processed under FOIA.” Then, in Section 2.3.11.2.2, “The Freedom of Information Act … require[s] NIH to release certain grant documents and records requested by members of the public, regardless of the intended use of the information. These policies and regulations apply to information in the possession of NIH.”

So, there’s an initial question to be answered: Does FOIA actually require this sort of data (e.g., proposals and peer review scores) to be made public? Yes, at least some of the time.

***

FOIA generally requires government data to be provided to the public, with several exceptions—three of which are usually mentioned here:

Exception 4: Trade secrets and commercial/financial information that is obtained from a person and that is “privileged or confidential.”

Exception 5: Inter-agency or intra-agency “memorandums or letters that would not be available by law to a party other than an agency in litigation with the agency”

Exception 6: “Personnel and medical files and similar files the disclosure of which would constitute a clearly unwarranted invasion of personal privacy.”

Also worth noting: FOIA’s exceptions just mean that an agency isn’t required to release the information in question, but the agency may still have the discretion to do so. See Chrysler Corp. v. Brown, 441 U.S. 281, 291 n. 11 (1979) (noting that FOIA exemption 4 is “an exception to the disclosure mandate of the FOIA, and not a limitation on agency discretion”).

NIH explains which types of information it will release in response to a FOIA request, and which types it will not (NIH Grants Policy Statement, Section 2.3.11.2.2):

In other words, the NIH does release funded applications pursuant to a FOIA request, along with progress reports and the like. But it won’t release information about pending or unfunded applications, nor will it release site visit reports or peer review statements.

The NIH doesn’t explain why it has this interpretation of FOIA. Indeed, you might note that its refusal to release unfunded applications and peer review scores (which are not FOIA exceptions) is separate and apart from its refusal to release “trade secrets,” “valuable commercial rights,” or other “competitive” information (which would be FOIA exceptions).

Let’s consider the FOIA exceptions in turn:

Exception 4: Trade secrets and commercial/financial information that is obtained from a person and that is “privileged or confidential.”

Some might argue that scientific research proposals should be considered as a “trade secret” that is “privileged or confidential.” After all, if a researcher thinks up a new idea or technique to explore in future research, it might be unfair if someone else got access to that unfunded idea, and was somehow able to scoop the initial research.

This is not a correct interpretation of FOIA, however. Not all proposals are a trade secret with proprietary information, and not even NIH thinks so. Applicants have to go out of their way to specifically identify which information might be proprietary. The NIH Grants Policy Statement notes that “applicants are instructed to identify proprietary information at the time of submission [NOTE: NOT AT THE TIME OF APPROVAL] of an application. . . . If an applicant fails to identify proprietary information at the time of submission as instructed in the application guide, a significant substantive justification will be required to withhold the information if requested under FOIA.”

In other words, even by NIH’s own interpretation of FOIA, grant proposals aren’t “proprietary information” unless the scholar provides some “significant substantive justification.”

As well, under the limited caselaw available, grant proposals and progress reports are not protected by Exception 4 of FOIA. In Washington Research Project, Inc. v. Dept. of Health, Education and Welfare, 504 F.2d 238 (D.C. Cir. 1974), the Washington Research Project sued to get access to “eleven specifically identified research projects that had been approved and funded by the National Institute of Mental Health.” This information included the grant application, a site visit report from the agency, and a summary report on the application.

One issue was whether the “research designs” were trade secrets or commercial information, in that “ideas are a researcher’s stock-in-trade.”

The court ridiculed this claim: “the government has been at some pains to argue that biomedical researchers are really a mean-spirited lot who pursue self-interest as ruthlessly as the Barbary pirates did in their own chosen field. . . . It is clear enough that a non-commercial scientist’s research design is not literally a trade secret or item of commercial information, for it defies common sense to pretend that the scientist is engaged in trade or commerce.”

The court therefore held that “research designs submitted in grant applications are not exempt from disclosure under the Act.” Moreover, “this holding extends to all types of applications — initial, continuation, supplemental, and renewal — and to progress reports made by grantees as part of the last three kinds of applications.”

This case has been reaffirmed more recently in Physicians Committee for Responsible Medicine v. National Institutes of Health, 326 F. Supp. 2d 19 (D.D.C. 2004). In that case, a group of animal welfare activists sued NIH over a “research project that involves giving amphetamines to cats infected with the feline equivalent of HIV.” The court did not agree with NIH’s claim that the information was a trade secret, or that it was confidential commercial information. While the researcher in question (a Dr. Podell) had claimed he was trying to develop potential treatments for AIDS, the court said, “He was not involved in trade or commerce when his research design was developed. . . . The fact that Dr. Podell was engaged in research for the university renders the possibility of a trade interest in his research design remote.”

Indeed, think about the disparity between how the NIH treats unfunded and funded proposals. When a proposal has been funded, NIH can release it under FOIA, whereafter everything can be broadcast to the world (with only a few exceptions). Here's an example of such an application that the NIH has posted publicly. As you can see, the NIH does black out occasional pieces of information, such as phone numbers and addresses. But the rest of the proposal isn't sensitive or confidential at all: it’s a description of proposed work on the innate immune system, a bunch of citations, a description of the lab space, and mostly people’s long CVs (which are usually publicly posted on their own websites anyway).

But the thing to note is that unfunded and funded proposals have literally the same content. No one retroactively changes the information in a proposal that happens to get funded. Occasionally, some information might be proprietary or private, but that information is the same both before and after funding.

So, in legal terms, there is no reason to treat unfunded proposals and funded proposals differently. Proprietary information might be funded (and need to be protected), but non-proprietary information often goes unfunded and doesn’t qualify for legal protection just because it was unfunded. FOIA in this case is about proprietary vs. non-proprietary, not funded vs. unfunded.

Why do NIH and NSF nonetheless treat this kind of information as sensitive/confidential/proprietary just because the proposal was unfunded? Sure, some scientists don't want to get scooped or lose intellectual property before being able to actually do a planned line of work, and therefore don’t want their unfunded proposals publicly available. But that doesn’t mean those proposals fit the legal definition of “trade secret” under FOIA. A proposal can be a trade secret whether funded or unfunded, and it can be a non-trade secret in either case as well. Funded-vs-unfunded and trade-secret-vs-not-trade-secret are two completely orthogonal issues. 

In short, under longstanding caselaw, NIH grant applications and progress reports cannot be generally treated as “trade secrets” that are “private or confidential" (unless identified and approved as such ahead of time). Neither FOIA itself nor any caselaw that I could find even hints that whether a proposal is funded or not has anything to do with whether the information is a trade secret or confidential business information.

Exception 5: Inter-agency or intra-agency “memorandums or letters that would not be available by law to a party other than an agency in litigation with the agency”

Peer review scores and similar data are typically treated as falling under Exception 5 to FOIA. That is, a number of cases have interpreted the so-called “deliberative process privilege” as covering advice from peer reviewers. See Formaldehyde Inst. v. Dep't of Health & Human Servs., 889 F.2d 1118, 1121 (D.C. Cir. 1989); Judicial Watch, Inc. v. U.S. Dept. of Commerce, No. 15-cv-2088 (D.D.C. Aug. 21, 2017); Washington Research Project, Inc. v. Dept. of Health, Education and Welfare, 504 F.2d 238 (D.C. Cir. 1974) (discussed above).

However, as discussed below, peer reviewer ratings and comments are hardly all that confidential (these, after all, are votes on how to distribute government funds). NIH/NSF/etc. could perfectly well work to anonymize and release these data to independent scholars even if not strictly required to do so by FOIA. Remember, FOIA exemptions just mean that the agency isn’t required to release the data to anyone, not that the agency is forbidden to release the data in its discretion.

There is a closely-related legal issue from a law that basically copies much of FOIA’S language, namely the Government in the Sunshine Act. HHS regulations refer to that Act in saying that meetings of NIH study sections are “closed to the public.” See 42 C.F.R. § 52h.6:

“Meetings of peer review groups reviewing grant applications or contract proposals are closed to the public in accordance with sections 552b(c)(4) and 552b(c)(6) of the Government in the Sunshine Act, as amended (5 U.S.C. 552b(c)(4) and 552b(c)(6)).”

What do those sections of the Government in the Sunshine Act say? That government meetings and deliberations have to be open to the public, with some exceptions that are basically copied from FOIA. I.e., government meetings need not be open and transparent if they involve “trade secrets and commercial or financial information obtained from a person and privileged and confidential“ ((c)(4)), or if they “disclose information of a personal nature where disclosure would constitute a clearly unwarranted invasion of personal privacy“ ((c)(6)).

But while peer review discussions could logically be treated as part of the “deliberative process,” it’s not clear why they would be automatically treated as involving “trade secrets” that are “privileged and confidential” information, let alone information that creates a “clearly unwarranted invasion of personal privacy” akin to private medical records. I can't find any relevant caselaw or legal articles explaining why concepts like “clearly unwarranted invasion of personal privacy” or “trade secrets” that are “privileged and confidential” would apply to the NIH's peer review process in all cases.

In other words, the HHS regulations purport to implement the Government in the Sunshine Act, but in fact are considerably more restrictive and conservative than what the law actually says. 

As an ex-lawyer, I understand the impulse to opt for the most restrictive and conservative interpretation possible, because the goal is to protect the agency at all costs. But that doesn’t mean it’s the right interpretation if one has the larger public interest in mind. The White House could issue an Executive Order requiring HHS (or any other agency with a similarly restrictive interpretation) to re-examine the interplay between the Government in the Sunshine Act, FOIA, scientific peer review, and the need for outside researchers to engage in research and evaluation pursuant to the Evidence Act of 2018. 

Exception 6: “Personnel and medical files and similar files the disclosure of which would constitute a clearly unwarranted invasion of personal privacy.”

Finally, there’s Exception 6. The main cases here held that despite FOIA Exception 6, NIH was required to release information about applicants who went unfunded!

In Kurzon v. Dept. of Health and Human Services, 649 F.2d 65 (1st Cir. 1981), George Kurzon “wanted to test his theory that the peer review method by which the National Institutes of Health (NIH) evaluate grant applications is biased against unorthodox proposals.” After unsuccessfully trying to get the data, he filed a FOIA lawsuit asking that the National Cancer Institute turn over “names and addresses of unsuccessful applicants for research grants.”

The lower court held that NIH didn’t have to turn over data, because of FOIA Exemption 6: “personnel and medical files and similar files the disclosure of which would constitute a clearly unwarranted invasion of personal privacy.” The court thought that this information “would be a serious unwarranted invasion of privacy and might reflect opinions about the competence of the applicant or his professional qualifications.”

The First Circuit disagreed and overturned the lower court’s decision. It noted that under most cases, the sorts of files at issue in Exemption 6 had “intimate details” of a “highly personal nature,” and that the lower court shouldn’t have merely asked whether the information was “personal” but whether it was “of the same magnitude—as highly personal or as intimate in nature—as that at stake in personnel and medical records.”

But here, the court said, Kurzon was just seeking names and addresses of rejected applicants—i.e., only “slight informational content,” and the loss of privacy would be “minimal.” Nor would there be a risk of embarrassment, since “twice as many applications are rejected as are not.” (That figure would be higher today.)

The First Circuit also emphasized the public nature of this data:

“Finally, federal grant applicants cannot reasonably expect that their efforts to secure government funds, especially in a field so much in the public eye as cancer research, will remain purely private matters. There is an obvious public element to the process and the results, as recognized in the NIH practice of releasing both the applications and identities of funded grant applicants.”

This makes perfect sense. It is unreasonable to ask for money from the government and the taxpayers in order to support your research, and yet also demand that everything about that process remain confidential.

The Kurzon case did allude to the possibility of a “promise of anonymity” on the part of NIH. The court held, however, that there “was no such promise,” and that “the best the government can do is to assert a general implied promise of confidentiality based on its policy statement, published in the Federal Register, that ‘[i]nitial research or [a] research training grant application on which award is not made’ is ‘generally not available’ to the public.” Kurzon, 649 F.2d at 69-70 (quoting 45 C.F.R. Part 5, App. (1980)).

Moreover, even if an agency (such as NIH) promised confidentiality to applicants, that cannot be a good reason to override FOIA. Otherwise, agencies could always evade future FOIA requirements just by unilaterally promising to keep certain documents secret.

In short, the only federal appellate case on this issue held that data on unfunded applications must be released under FOIA.2

Other cases have followed this precedent. For example, here’s a 2001 case in which the federal district court for New Hampshire required the National Institute of Mental Health to release the names and addresses of unfunded researchers, again to George Kurzon (the aspiring metascience researcher who wanted to ask unfunded researchers about whether they thought peer review “does not sufficiently recognize the value of innovative cutting-edge research”). In that case, the court noted that HHS’s argument that “confidentiality permits scientists to avoid or hide their application records with impunity is not a persuasive argument.”

Why don’t NIH and NSF take these cases more seriously in their daily dealings with outside researchers? That’s a good question. The White House should demand that agencies regularly work with outside researchers to let them do research and evaluation on the content and disposition of unfunded proposals, without anyone needing to file future FOIA lawsuits.

B.    The Privacy Act

Whenever NIH and NSF are not strictly required to release data to the public under FOIA, they typically find reasons to fall back on the Privacy Act, which protects individual privacy as to data held by the government. For example, I asked both NIH and NSF lawyers for any memo or explanation as to why they restrict access to data on unfunded proposals and peer review. I never heard back from NIH, but NSF personnel reached out to say the following:

“Most of the data that NSF collects at the individual person-level as part of the merit review process is protected by the Privacy Act, which generally prohibits disclosure without consent of the individual. Like other federal agencies, NSF publishes Privacy Act System of Records Notices (SORNs) in the Federal Register that explain what Privacy Act protected data the agency is collecting and how it may be disclosed. You can find links to all of NSF’s SORNs here: https://www.nsf.gov/privacy/.”

In another email, the NSF lawyers told me:

The statutory language of the Privacy Act describes a “system of records” that is covered by the Act as “…a group of any records under the control of any agency from which information is retrieved by the name of the individual or by some identifying number, symbol, or other identifying particular assigned to the individual…” 5 U.S.C. 552(a)(5).

Because NSF routinely retrieves proposal information by the name of the Principal Investigator who submitted the proposal, the proposals are maintained in a system of records covered by the Privacy Act. NSF peer reviewer information is handled similarly. In fact, NSF has prevailed in litigation on this issue: https://www.nsf.gov/news/news_summ.jsp?cntn_id=100876. NSF does not publicly disclose data sets where there is a reasonable risk of re-identification of individuals.

As an initial matter, does the Privacy Act actually require NSF or NIH to protect any and all information relating to unfunded proposals and/or peer review scores?

No.

The baseline requirement is that individuals have a right to expect the government to keep their personal information private, with a number of exceptions. That is: “No agency shall disclose any record . . . by any means of communication to any person, or to another agency, except pursuant to a written request by, or with the prior written consent of, the individual to whom the record pertains . . . .” 5 U.S.C. § 552a(b).

But there are two issues here: 1) Exactly what information gets Privacy Act protection? 2) What exceptions are there?

First, let’s talk about what sort of information is at issue. The law talks about “records,” a term defined as “any item, collection, or grouping of information about an individual that is maintained by an agency, including, but not limited to, his education, financial transactions, medical history, and criminal or employment history and that contains his name, or the identifying number, symbol, or other identifying particular assigned to the individual, such as a finger or voice print or a photograph.” 5 U.S.C. § 552a(a)(4).

Right from the start, you may notice that “records” have to contain someone’s “name” or some identifying information. If you took the names and other identifying information out of an NIH/NSF proposal or its peer review scores, it wouldn’t be a “record” under the Privacy Act in the first place!

Another point is to note the kind of records at issue – someone’s education, financial status, medical history, criminal history, etc. All of this amounts to personal information (in some cases, highly personal information). The mere fact of applying for an NIH/NSF grant, or the peer review scores given to the application, don’t resemble someone’s medical or financial history in terms of the privacy interests involved.

Yes, the statute says that “records” include but are “not limited to” those examples. But by traditional principles of statutory construction, words in a list typically are given a related meaning. The list isn’t for nothing—it serves to demonstrate the kinds of things at issue.

For example, if a statute refers to a park being off limit for “vehicles, including but not limited to cars, trucks, motorcycles, and electric scooters,” that list is telling us something about the word “vehicles.” A child’s tricycle would probably not be treated as a “vehicle” to be excluded from the park. After all, the purpose of the list seems to be excluding motorized vehicles that might pose a danger to children or pedestrians. A child’s tricycle isn’t anything like that at all, even if it might technically be a “vehicle.” So too, data on peer review scores for a research proposal does not resemble any of the statutory examples of personal data.

But to clinch the case, even if the definition of “records” stretches broadly enough to include proposals and peer review scores, there’s a key exception:

Agencies ARE allowed to disclose records to a “recipient who has provided the agency with advance adequate written assurance that the record will be used solely as a statistical research or reporting record, and the record is to be transferred in a form that is not individually identifiable.” 5 U.S.C. § 552a(b)(5).

In other words, if an outside researcher is merely looking to perform “statistical research” on data that “is not individually identifiable,” then the Privacy Act provides no reason for NIH or NSF to decline to provide that data.

There might be a debate about what “individually identifiable” means, and some people might argue that pretty much any information is individually identifiable. In technical, statistical terms, that might be true. If there’s a grant application on subject X submitted at a particular time by someone at a university of a particular size, etc., etc., etc., a clever data scientist or an AI tool could probably identify the person who submitted that application.

That said, the law doesn’t define “individually identifiable” in such an expansive fashion, and we shouldn’t worry about that sort of sophisticated effort at reidentification. Simply put, no political scientist, economist or policy scholar would undertake the trouble of getting approval to work with internal NIH or NSF data, all for the purpose of an underhanded attempt to figure out whether it was John Doe at the University of Wisconsin or Jane Doe at UCLA who submitted a particular application as to cell biology. Outside scholars are interested in broader macro questions, not in reidentifying anyone (as if such reidentification would be harmful in any event).  

Moreover, a general rule for statutory interpretation is that statutory provisions should not be interpreted so as to be a nullity. If “individually identifiable” was interpreted so broadly as to apply to virtually any release of data under any circumstances, then there would be no point for Congress to have included an exception for “statistical research” in the first place. Congress clearly intended that “statistical research” should be allowed, and that necessarily means that Congress did not intend for all data to be treated as “individually identifiable” so as to preclude the possibility of all statistical research.

Now, what about the NSF’s rationale as provided to me? I think it doesn’t quite add up. As a reminder, NSF told me that under the Privacy Act, a “system of records” is anything where information “is retrieved by the name of the individual,” 5 U.S.C. 552(a)(5), and that NSF “routinely retrieves proposal information by the name of the Principal Investigator.”

But this merely shows that a “system of records” exists at some point. It does nothing to show why that information remains a protected “record” if the individual names are stripped out, let alone if the data is used for “statistical research” under the Privacy Act’s exception. To use a legal cliché, the tail would be wagging the dog if we allowed the mere fact that an agency stores the person’s name somewhere to make it impossible to release anonymized data in ways that the Privacy Act expressly allows (and that FOIA often requires).

Moreover, NSF claimed to me that “NSF peer reviewer information is handled similarly,” and that “NSF has prevailed in litigation on this issue.” This isn’t quite right. Here, the NSF is referring to Henke v. U.S. Department of Commerce, 83 F.3d 1445 (D.C. Cir. 1996), a case in which someone who failed to win an NSF grant as well as a Dept. of Commerce grant (Henke) sued NSF demanding to know the identities of the peer reviewers who rejected the proposal.

The court noted that under the Privacy Act, individuals (such as applicants) have the right to see information kept about themselves, but there is an exception for “investigatory material“ related to grant eligibility if provided “under an express promise that the identity of the source would be held in confidence.” The court also noted that NSF had told reviewers that “the identity of reviewers will be kept confidential to the maximum extent possible.”

As the court said, the identity of peer reviewers needs to be confidential so that they will be candid without fear of repercussion. That is, they need the ability to be honest without worrying that they might “offend a colleague whom they may like as a person, or hope to work with in the future.”

All of that may be true, but has nothing to do with the case of independent scholars who wish to see anonymized peer review scores so as to evaluate how agencies are funding research. It is hard to see how such research would be threatening to peer reviewers, or cause them to temper their true beliefs. Moreover, the statutory exception applies “only to the extent that the disclosure of such material would reveal the identity of a source . . .” If no peer reviewer identity is revealed, then that term of the Privacy Act doesn’t apply in the first place.

Indeed, the NSF’s own official guidance on data regarding proposals (including peer review and later reports) notes that “information may be disclosed to” researchers who “carry out statistical studies for or otherwise assist NSF with program management, evaluation, or reporting. . . . Disclosures are made only after scrutiny of research protocols and with appropriate controls. The results of such studies are statistical in nature and do not identify individuals.”

In short, there is no statutory reason under the Privacy Act (or the accompanying caselaw) for NIH or NSF to have a blanket refusal to provide deidentified data on proposals and peer review scores to outside researchers who are merely trying to do “statistical research.” That position is, in fact, contrary to both the plain text and the spirit of the law.

As for the NIH, that agency’s policy (or System of Records Notice) under the Privacy Act seems oddly narrow. When it comes to data on proposals (including peer review scores, other reports, etc.), the NIH says that it will provide data:

To a party for a research purpose when NIH:

(A) Has determined that the use or disclosure does not violate legal or policy limitations under which the record was provided, collected, or obtained;

(B) has determined that the research purpose (1) cannot be reasonably accomplished unless the record is provided in individually identifiable form, and (2) warrants the risk to the privacy of the individual;

(C) has required the recipient to (1) establish reasonable administrative, technical, and physical safeguards to prevent unauthorized use or disclosure of the record, (2) remove or destroy the information that identifies the individual at the earliest time at which removal or destruction can be accomplished consistent with the purpose of the research project, unless the recipient has presented adequate justification of the research, and (3) makes no further use or disclosure of the record except when required by law, and reports results of the research in de-identified or aggregate form . . . .

As the bolded phrases indicate, this paragraph seems to be talking only about individually identifiable data. And in that case, it might make sense to be fairly restrictive.

But as discussed above, when identifiers are stripped out of the data, the data is arguably not subject to the Privacy Act at all. At a minimum, the data would be subject to an exception if the researcher merely provides “advance adequate written assurance that the record will be used solely as a statistical research or reporting record.”

Thus, if the NIH takes out identifiable information, it would be completely superfluous to ask researchers to prove that their research “cannot be reasonably accomplished unless the record is provided in individually identifiable form.” Those researchers aren’t even asking for the individually identifiable form of the data in the first place!

In other words, it is quite an odd oversight that the NIH’s Privacy Act notification doesn’t even contemplate releasing non-identifiable information under the “statistical research” exception from the Privacy Act itself.

Conclusion: The White House should direct NIH and NSF (and perhaps other research-funding agencies as well) to rewrite their policies to be more in accordance with the Privacy Act, which directly allows sharing of internal data for “statistical research.”

Everything Is Too Dichotomous: We Need a Better Legal Regime for Letting Scholars Work With Slightly-Private Internal Data

As we’ve seen above, agencies regularly interpret the law in an overly conservative fashion—they ignore FOIA cases that require the release of data on unfunded proposals, and they point to the Privacy Act as a reason never to release data even though the Privacy Act has a straightforward exception for “statistical research.” Even under current law, agencies can and should do much more to release data to independent researchers.

But part of a bigger problem is this: Our current legal regime is arguably too dichotomous or binary. If data is treated as subject to release under FOIA, then in theory it could be released to anyone at all, and could be posted on a public website. On the other hand, if data is treated as subject to a FOIA exception (and/or as protected under the Privacy Act), then no external researchers get the data at all.

That all-or-nothing approach doesn’t fit well with the kinds of data we are talking about. Consider peer review comments. We might not want the full text of reviews to be posted on a public website, at least not without consent— it might be embarrassing to applicants, and might compromise the integrity of the process. Same goes for unfunded applications, which scientists would often want to be kept private in case someone else scoops their idea.

That is why in some of the cases described above, agencies like NIH and NSF are struggling to figure out how to avoid FOIA—they don’t want to be put in the position of having to release all of this data to the entire world.

But outside researchers (e.g., economists and political scientists) aren’t engaged in the usual FOIA-type activity. Plastering information on a public website is not what these researchers want to do at all. Instead, they want to perform academic research that, if necessary, treats any given unfunded application or peer review score as confidential information, and that just analyzes overall questions such as, “Are more innovative proposals doing worse in peer review, and if so, why?”

In other words, these types of data are slightly private. They are not fully public, per se—you wouldn’t necessarily want the data to be published in fully identifiable form on the Internet. But neither are they fully private, in the sense that they need provable statistical anonymization (such as differential privacy) and accessibility only on computers walled off from the Internet.

The latter would be overkill. Remember, these materials all concern requests for public funding, and should by default be as freely and openly available as possible. As a federal appellate court mentioned above held:

“Federal grant applicants cannot reasonably expect that their efforts to secure government funds, especially in a field so much in the public eye as cancer research, will remain purely private matters. There is an obvious public element to the process and the results, as recognized in the NIH practice of releasing both the applications and identities of funded grant applicants.”

There are a number of existing options for handling this kind of slightly private data. In many or most cases, it is likely sufficient just for NIH and NSF to require IRB approval and a signed confidentiality agreement that states, “No personally identifiable information will be published.”

Another alternative is for NSF and NIH to engage with the Institute for Research on Innovation in Science (IRIS), hosted at the University of Michigan. IRIS is a coalition of major universities that contribute “data on 580,000 funded awards worth $192 billion, payments to more than 1.2 million vendors ($35 billion), and wages to approximately 985,000 employees.” It provides access to that data on a privacy-protected basis, with a virtual data enclave supplied to researchers who have IRB approval and who also get approval for their research plan. NIH and NSF data would be a great resource to integrate with the university-level data.

Yet another alternative is that NIH and NSF could try to work with the OMB’s new data resource for “restricted microdata from federal statistical agencies,” namely, ResearchDataGov.org. This website is a platform for getting access to confidential data from, among others, the Census Bureau, the Bureau of Labor Statistics, and the IRS Statistics of Income Division. This was the result of a large, federal-wide effort to make a streamlined platform for accessing such confidential data from over 15 agencies. NIH and NSF could easily piggyback on this existing work (although their data might be more useful if combined with IRIS data above).

In any event, even if you think that NIH/NSF proposals and peer review scores need some sort of top-level protection for confidentiality – which is illogical and contrary to the caselaw on the issue – there’s no reason not to use IRIS or other existing systems to allow outside researchers to do their independent evaluations.

But in any event, it might help if Congress passed a version of “Evidence Act 2.0.” Such an Act could help clarify the options for how federal agencies should handle slightly private data that needs to be widely accessible to independent researchers without going as far as FOIA. We’d have more and better research on how NIH and NSF perform with upwards of $59 billion a year.

FUTURE ACTIONS

The White House:

The White House could, by executive order, ask federal agencies that fund research to:

• “Make data open by default” (see section 202(c) here) to independent researchers with approval by a qualified IRB, including data on internal operations such as peer review scores, disposition of research proposals (included unfunded proposals), and perhaps even analyses by program officers and other agency officials (if anonymized);

• Stop invoking FOIA Exception 4 (trade secrets and commercial/financial information that is obtained from a person and that is “privileged or confidential”) as to any of the above information, unless that exception has been specifically invoked and demonstrated ahead of time by the researcher in question;

• Re-examine the Government in the Sunshine Act and any rules thereunder, so as to enable outside researchers to have full access to any meetings where decisions are made on research funding;

• Reassess all data access policies so as to allow to much more streamlined research and evaluation pursuant to the Evidence Act of 2018;

• Rewrite their policies (as necessary) to be more in accordance with the Privacy Act’s express allowance for “statistical research.”

Congress:

Congress could enact legislation like the Grant Reform and NIH Transparency (GRANT) Act, a bill introduced by Rep. Huizenga. This bill would require NIH to take substantial steps to share its internal data with qualified external researchers. This legislation could expanded to all federal agencies that fund scientific research.

Much thanks to Katy Rother, Julia Lane, and an anonymous White House official for their comments and advice, although all opinions in this brief remain entirely my own.

Last month, the Academic Freedom Alliance issued guidance highlighting the threat that Institutional Review Boards (IRBs) pose to the freedom to conduct research at colleges and universities.

If you’re outside academia, you may not have even heard of IRBs (sometimes called Human Subjects committees). But they wield consequential power over what research is pursued in our truth-seeking institutions, and as a result the knowledge the wider public gets to have and benefit from.

Through overreach and mission creep, the arbitrary and often viewpoint-discriminatory processes employed by IRBs can too easily be used to suppress unpopular research, preventing consequential and potentially groundbreaking scientific endeavors from getting off the ground.

What is an IRB, and why do they exist?

An IRB is a committee at a college or university that reviews and approves any kind of research involving human subjects. The idea is straightforward: the IRB wants to make sure that nobody is harmed and that participants know what they’re signing up for when participating in a study.

If this sounds reasonable, that’s because it is. In fact, IRBs were established by federal directives in the U.S. after major ethical abuses in research. One of the most infamous was the untreated syphilis study at Tuskegee between 1932 and 1972, in which the U.S. government studied the effects of syphilis on black men for decades without treating them — or even telling them what disease they had.

In 1979, the federal government issued the Belmont Report, which laid out three core principles for ethical research:

1. Respect for persons: People must give informed consent.

2. Beneficence: The benefits of research must be weighed against the risks.

3. Justice: The burdens and benefits of research should be shared fairly.

These reforms also drew on earlier research ethics principles reflected in the Nuremberg Code, which were written in the aftermath of the Holocaust and in response to the medical atrocities committed by Nazi doctors.

IRBs, then, became an oversight mechanism for enforcing these principles at the local institutional level. Today, any institution conducting federally regulated human subjects research is required to have an IRB, and many other institutions voluntarily maintain similar review bodies.

These safeguards serve a vital purpose. No one wants a return to the days when researchers could experiment on people without their knowledge or consent, or inflict lasting harm on participants just to attain some desired research data.

So, what went wrong?

Why IRBs have become a problem

Like many well-intentioned oversight mechanisms, over time a significant number of IRBs have become object lessons in scope creep and unchecked authority.

What started as a system for protecting people from genuine physical and psychological harm has, at many institutions, ballooned into a bureaucratic apparatus that can delay, distort, or even derail research — sometimes for reasons that have nothing to do with participant safety.

For example, many IRBs now have authority over research that poses no meaningful risk to participants. Simple surveys, or in some cases even the analysis of public records, routinely face scrutiny similar to that received by far riskier research. Aspects of studies that wouldn’t have warranted a second look a decade ago are now subject to intense review at our major institutions. In effect, bureaucratic hurdles are slowing or preventing research that has no meaningful risk of harm.

This has led some to argue that IRBs “subject researchers to petty tyranny,” and others to claim that they “have become nationwide instruments for implementing censorship” because of their ability to operate as bureaucratic gatekeepers: IRBs have the power to permit research that meets their satisfaction, and suppress data and prohibit publication for research that doesn’t. This is particularly true on hot-button, controversial issues.

Funnily enough, the expanding scope of IRBs over time may in part be explainable by more ethical research being conducted at colleges and universities.

In one of several studies, psychology professor David Levari and his colleagues asked participants to play the role of an IRB reviewer. Participants read a series of research proposals and then decided whether each should be approved or rejected. With the obvious caveat that these clearly weren’t trained IRB reviewers, Levari found that as the prevalence of genuinely unethical research proposals decreased over time, reviewers did not start to rate fewer proposals as unethical. Rather, they started rejecting ethically ambiguous proposals that they would have approved earlier on. In other words, even as research practices improve, IRBs may keep finding problems by unconsciously lowering the bar for what counts as problematic.

People wielding hammers will always find more nails.

Needlessly expanding scope isn’t the only problem, however. IRBs can also be weaponized to suppress or stymie research on unpopular or hotly debated topics. As former IRB chairs Jessica Hehman and Catherine Salmon recount, there are many well-documented cases where IRB review processes have been used to thwart research for ideological reasons — or to simply protect an institution’s brand.

Consider the case of Elizabeth Loftus and Melvin Guyer. In the late 1990s, at the peak of the “Memory Wars” — the sudden popularity and then debate over the eventually-discredited practice of recovered memory therapy — Loftus and Guyer embarked on a pivotal investigation into a Jane Doe case they believed overstated proof of recovered traumatic memory. In turn, their articles and investigation provoked prolonged institutional retaliation.

A central actor in the retaliation, it turned out, was the University of Michigan IRB, whose chair drafted a confidential memo that was leveraged to target both Loftus and Guyar. Both researchers were eventually cleared, but in the three years of investigation they — and in particular, Loftus — were subjected to the seizure of research materials, lengthy misconduct investigations, and threats of professional sanction. The shadow of these events still lingers decades later.

IRBs in some cases also appear to look out for themselves and their institutions more than those they are actually charged with protecting. In studies that have surveyed the researchers themselves — federally-funded principle investigators in one, criminology researchers in others — about their views of IRBs, many reported perceiving that IRBs are more concerned about protecting themselves or the university from liability than actually protecting human subjects.

The problem isn’t altogether new either. Back in 1985, psychology professor Stephen Ceci and colleagues found tentative evidence that IRBs were more likely to block research on politically sensitive topics than similar research on less controversial topics. They argued that some IRBs were reacting not just to participant risk, but also to worries about what the research might imply politically or socially.

And that’s where the academic freedom issues begin: IRBs began drifting beyond protecting participants and toward policing ideas.

FIRE and the AFA’s proposed solutions to IRB overreach

FIRE researchers have also seen or personally experienced IRB overreach firsthand.

In one case, there was an almost year-long delay by an IRB for a project looking at public records. We have also seen delays or punitive audits of benign survey research which involved asking faculty about their views on free speech and related topics.

FIRE researchers have also been on the receiving end of legal threats from faculty, as well as frivolous ethics complaints submitted by faculty to their IRBs. Just to give you a sense of how far IRB overreach can extend, these complaints were filed during the data collection phase, before anything was even analyzed. The goal was to suppress research — specifically, research questions — that these faculty members disagreed with, before any conclusions or findings could be reported.

If these patterns sound familiar, they should. IRB overreach is an integral part of what Greg and his Canceling of the American Mind co-author Rikki Schlott have referred to as the Conformity Gauntlet: the long series of ideological hurdles that independent-minded academics face at every stage of their careers, from graduate school admissions to tenure review. Central players in the Conformity Gauntlet, of course, are DEI statements, bias response teams, and secret disciplinary hearings. As we have seen, IRBs, too, can be captured by ideology and used as a checkpoint where nonconformist research or researchers can be pressured or silenced. No single hurdle is insurmountable, but when combined these barriers powerfully incentivize conformity and disincentivize open and fearless inquiry — the very thing universities should exist to protect.

FIRE has previously endorsed IRB reforms aimed at protecting academic freedom by guarding against this type of overreach. For example, when we discussed potential ways the National Institutes of Health could protect academic freedom through its grant agreements, we wrote:

NIH could help prevent the abuse of Institutional Review Boards. When IRB review is appropriate for an NIH-funded project, NIH could require that review be limited to the standards laid out in the gold-standard Belmont Report. Additionally, it could create a reporting system for abuse of IRB processes to suppress, or delay beyond reasonable timeframes, ethical research, or violate academic freedom.

The AFA’s new guidance goes further than this, endorsing ten reforms from a set of principles called the Mudd Code. Many of these proposals are similar to what FIRE has recommended in the past, and include things like transparency requirements, which would make it easier to identify if viewpoint discrimination had taken place; clarity on risk thresholds; clarification that IRB approval is not necessary for the re-evaluation of existing science; and the addition of a science advocate to the composition of IRBs themselves.

These reforms are aimed at preserving the important role of ensuring safe and ethical research practices, while also guarding against the kind of ideological and bureaucratic overreach that can too easily be used to suppress unpopular research.

While some of the AFA and Mudd Code recommendations fall outside FIRE’s narrow scope as a free-speech watchdog, we applaud them for bringing light to the threat that IRBs can pose to academic freedom. We also commend them for proposing concrete, principled reforms to address it.

Our institutions of higher education are places where difficult, groundbreaking, and important research should be done. That requires asking questions and dealing with subjects that might be uncomfortable and controversial. It also means grappling with results that may be politically or ideologically inconvenient. The truth remains true regardless of our concerns, and if we fail to discover it we will be compromising much more than we think.

Eliah is a Professor of Medicine and Design at UC San Diego’s Division of Infectious Diseases & Global Public Health, and the Design Lab. He directs the Center for Health Design, the CDC funded Resilient Shield, and chairs the Lancet Commission on U.S. Societal Resilience in a Global Pandemic Age.

***

In 2019, Patrick Collison and Tyler Cowen published an Atlantic essay that quietly launched a vital new field. Progress Studies, as they named it, would document the nature of scientific, technological, and economic progress; asking why it sometimes flourishes, why it sometimes stalls. Seven years later, we believe that project is necessary but no longer sufficient. Understanding progress isn’t enough. We need to engineer it.

Progress Engineering, in our vision, will be a discipline devoted deliberately to the practice and principles of designing systems, organizations, and institutions through which innovation moves. It begins where Progress Studies ends: not with the question of what progress looks like, but how we build the conditions that make it possible, reliable, and just.

The difference matters enormously. The great innovations from Gutenberg’s press to steam power to the assembly line to IT and now AI have all involved a massive increase in the returns to scale in key industries, and each of them led to social changes that well-meaning contemporaries struggled to theorize over and to shape for the benefit of humanity.

Yes, there is a role for Hayek-Schumpeter spontaneous order and creative destruction, for innovation fueled by nearly unplanned experimentation. But these past waves of innovation all had their own progress engineers who led ever-larger firms, who shaped ever-growing legal systems, and who created often-fragile political consensuses to create innovation-friendly environments.

This common thread–a rise in the efficient scale of key firms and the need to not-manage-but-engineer the social response to these rapid, visible, and highly-productive increases in economic scale–offers a hope that Progress Engineers can learn from the past to uncover principles that raise the probability of successful, pro-human progress.

What is Progress Engineering?

The field of progress studies creates interdisciplinary research into the nature of human progress, along with policy work that reduces barriers to progress: the research helps people to visualize progress while the policy work encourages and permits progress. We see Progress Engineering as adding a critical third component: a body of practical knowledge–and knowledgeable practice–that goes beyond observing, understanding, or encouraging progress, toward ethically engineering and humanely employing progress itself.

More akin to design thinking, or perhaps even to Buddhist practice, Progress Engineering must be understood as a skill set combined with a mindset. The skill set contains the practical knowledge: but the knowledgeable practice comes from a mindset and experience: rule-shaped but never rule-driven.

Engineering is the right metaphor for this combination. By analogy, Nobel-Laureate Eric Maskin describes mechanism design as the “engineering” side of economic theory, taking resource constraints and self-interested human behavior as given, defining desired outcomes, and then building institutions or incentives to achieve those outcomes to the greatest degree possible.

Similarly, Progress Engineering, taken seriously, helps people choose prudent goals for organizations, localities, and societies: goals that might range from scientific breakthroughs to economic inclusion to system-wide improvements in public health. Progress engineering then supplies the practical knowhow to shape and tend the complex mixes of incentive structures, cultural practices, and institutional frameworks needed to maximize the chances of accomplishing those goals.

This approach would have to combine insights from multiple domains and sectors, drawing on ethics, religion, policy, and law for objective functions and moral guardrails, on mechanism design in economics to navigate, on organizational science for fostering innovation culture, on decentralized science, human centered design, on codesign and service design methods that build in inclusion and stakeholder experience, and on ecology and political science for polycentric governance models that enable fruitful cooperation in harmony with the environment.

If it sounds like progress engineering is almost impossibly multidisciplinary, that’s because it is, just as progress studies already is, having already gone well beyond its origins in economic history to include all forms of engineering, the sociology of organizations, and cultural and philosophical studies of innovation, just to name a few of the fields that progress studies practitioners routinely draw upon. Yes, there will be limits to how much interdisciplinary knowledge experts in these twin fields can be expected to digest, but few fields of study can be ruled out in advance–and knowledgeable practice will convey the limits to multidisciplinarity.

Progress Engineers thus should start where mechanism design ended up. Mechanism design began as pure economic theory but found the need to incorporate lessons from behavioral economics, experimental economics, and computer science to create practical, efficient auctions. Over time, law and public choice also became part of the applied mechanism design toolkit. Progress Engineers should cut to the chase, embracing the obvious need for cross-cutting expertise when searching for the rules underlying humane progress.

Crucially, Progress Engineering is intentional and democratic, living in the actions of all, and working to better their circumstances. Rather than waiting for innovation to emerge serendipitously, it treats progress as a product of stakeholder design. History provides positive outliers that demonstrate this principle in action, from prizes that sparked new industries to corporate cultures that institutionalized continuous improvement. By examining these examples, we can discern the principles of Progress Engineering and see how to apply them to today’s grand challenges.

Progress by Design: Lessons from XPRIZE to COVID-19

1. Incentive Prizes and Engineered Breakthroughs. In 1996, entrepreneur Peter Diamandis launched the $10 million Ansari XPRIZE to spur private spaceflight. The prize created a clear goal and incentive for anyone to solve it. More than two dozen teams poured over $100 million of their own R&D spending in pursuit of the $10 million prize. In 2004, Mojave Aerospace Ventures won with SpaceShipOne. The contest catalyzed an industry. This is mechanism design in practice: the XPRIZE architects designed a competition that harnessed latent talent and capital, effectively engineering a breakthrough by aligning incentives with a bold goal.

2. Digital Infrastructure for Inclusive Innovation. Consider India Stack, a government-led digital public infrastructure comprising identity (Aadhaar), payments (UPI), and data-sharing protocols. This stack was deliberately engineered to be open-access and scalable. Over 1.2 billion Indians obtained a digital ID, giving previously undocumented people a verifiable identity. Using that foundation, India Stack widened access to finance in what was a cash-based economy. According to the IMF, this digital infrastructure is “revolutionizing access to finance” for millions. The architects of India’s digital infrastructure created technological building blocks that others could leverage for innovation. This exemplifies Progress Engineering at a national scale – policymakers designing public goods that unleash collective innovation.

3. Cultures of Continuous Improvement (The Toyota Way). Innovation isn’t only about moonshots; it can be baked into the daily fabric of an organization. Toyota’s leaders deliberately engineered the Toyota Production System (TPS) – a management system obsessed with eliminating waste and empowering workers to improve. Every employee is trained in kaizen, or continuous incremental improvement. Toyota designed an organizational system that harnesses the talents of many. This ethos has been emulated in industries from software to healthcare.

4. Rapid Collaboration in Crisis (COVID-19 Vaccines). The development of COVID-19 vaccines in 2020 compressed a 5–10 year process into about 11 months. This was the result of deliberate coordination, massive funding, and open collaboration across institutions. Scientists shared data, and new partnerships formed. Governments and nonprofits invested tens of billions in vaccine projects in parallel. Operation Warp Speed and advance purchase commitments removed financial risks. This exemplified Ostrom-style polycentric problem-solving. With the right structures – ample funding, agile regulatory coordination, open science – innovation accelerated. It was Progress Engineering under pressure.

5. Decentralized action. As HBS’s Linda Hill has argued, innovation is rarely solo genius; it is built upon intention and collective genius; and nowhere is that clearer than in the spontaneous organizations that emerge under duress. Both the Cajun Navy, which saved lives after Hurricane Katrina, and the Syrian White Helmets, saving lives in opposition-controlled Syria, started from the bottom up, combining local knowledge and local care, each later becoming more organized, more powerful, and more effective. Explaining the success of such organizations at the Progress Engineering level would require explaining the decentralized volunteer organizations that don’t exist, explaining, with a high degree of reliability, why some arise while others don’t. Those explanations, converted into practical action plans, would fill the gap between progress studies and Progress Engineering.

The Stakes: Why We Need Progress Engineering Now

If a combination of AI and robots makes much human labor irrelevant, it could mean a world of capitalists without customers, moving towards a class of AI- and robot-wielding oligarchs segmented from an increasingly impoverished populace. The climate could become more unstable, threatening food, commerce and transport. In that world, the environment and technological innovation both become enemies of progress for communities across the globe.

There is another path; one where the emergence of purposeful AI transforms bioengineering from academic endeavor to programming the natural world; a path to personalized medicines, abundant energy stores, robust crops, environmentally-friendly housing material. Physical AI and self assembling materials zero could the cost of production for housing, sundries and food. AI exploring science may unlock new truths that change our trajectory in the universe, and perhaps provide societies that are more secure, more inclusive, thriving with an abundance of resources.

Engineering this path forward, or even some yet-to-be-imagined better path, will be complex, and will require new ways of thinking, collaborating, and working, new ways of protecting ourselves from the power of our own technologies and our short memories for past atrocities in the name of progress. Like a child running with scissors, or a man sawing his own branch from a tree to gather fuel for the winter, we may be entering a period of accelerating transformation with little foresight and few clear plans for what comes next.

The coming changes bring the potential for global abundance and democratization of essential human needs alongside real risks of collapse of the longstanding environmental patterns and human structures that have underpinned our modern society. AGI and its consequences will require mindful responses from the public, from entrepreneurs, and from governments alike. That mindfulness will look like Progress Engineering: generalizing from past experience, drawing on first principles of human technological progress, and experimenting thoughtfully as problems arise. Regulatory sandboxes, incentive structures for socially beneficial innovation, and collaborative risk monitoring–by firms, nonprofits, and governments–are needed. Again, these are social design problems – Progress Engineering problems.

Finding realistic, politically palatable ways to systematically improve public health–really, public hygiene–is another critical goal for Progress Engineering. Not only do falling rates of vaccination and health surveillance pose risks to American and global public hygiene; more importantly, the rise of AGI will give bad actors small and large, domestic and foreign, greater potential to launch viral and bacterial bioweapons at our citizens. A prepared society will be a stronger society, a healthier society.

As a form of engineering, Progress Engineering will embody the fusion of broad principles, toy experiments, informed hunches, and practical tinkering central to all other forms of engineering. The engineer’s practical approach—“Does the boat float?” “Is the nation innovating more?” — gives a telos, a goal to Progress Engineering, but it will also need an ethos; forms of continuous tending to the moral consequences of progress, continuously reducing the risk that has runaway consequences or devolves into mere collections of facts, studies of the past, theories without applications. Fusing codified, rule-shaped techne with the local, tacit knowledge of metis: that is the path forward.

Progress can be engineered: indeed, progress has been methodically engineered already with varying degrees of success. By studying the Progress Engineers of the past, both those who succeeded and those who failed, future Progress Engineers will uncover principles to succeed robustly.

Moral Hazards and Ethical Guardrails

An essay such as this cannot be written without acknowledgment of past horrors that occurred in the name of improving society or bettering the condition of all of mankind. Nor can we understate the dangers of either weaponized science or well-intended social engineering. These dangers are real and their threats are certain. As such, we say in the strongest terms that these lessons must be deliberately integrated into any form of progress engineering. We cannot and should not stop technological and social change, but we can, as individuals and as a species, direct such change away from negative-sum rent-seeking kleptocratic oligarchy and toward a flourishing, bountiful good.

Conclusion

In 1861, the scientist William Barton Rogers proposed a new kind of research institution dedicated to practical knowledge. He founded the Massachusetts Institute of Technology, joining other reformers who realized that America’s ability to generate progress could be designed and improved. As Patrick Collison and Tyler Cowen² note in the Atlantic article that spurred the rise of progress studies, these 19th-century visionaries “engaged in a kind of conscious ‘progress engineering’” by restructuring institutions to spur innovation. Yet even today, progress itself remains under-engineered. Rogers’s MIT of the mid-19th century can serve as a model for the Progress Engineers of our dawning mid-21st century: not just collecting facts and publishing reports, but tinkering with purpose, experimenting without naivete.

Institutions are malleable, and the world’s people crave a sense that we can make progress in this new age. With deliberate design, we can make innovation more reliable and inclusive. It’s time to generalize past successes, be mindful of historic tragedies in the name of progress, and apply these lessons with intention and bravery to the complex, systemic challenges ahead. History shows that with the right frameworks, progress can be continuous and far-reaching. Let’s build those frameworks and then democratize the practice of engineering progress.

Sources:

1. Maskin, E. (2007). Mechanism Design: How to Implement Social Goals. Nobel Prize Lecture.

2. Collison, P. & Cowen, T. (2019). We Need a New Science of Progress. The Atlantic.

3. XPRIZE Foundation (2016). Ansari XPRIZE Overview.

4. Carrière-Swallow, Y., et al. (2021). Stacking up financial inclusion gains in India. IMF Finance & Development.

5. Hill, L. (2014). Collective Genius: The Art and Practice of Leading Innovation.

6. Ostrom, E. (2010). Beyond Markets and States: Polycentric Governance of Complex Systems. AER.

7. RAND Europe (2024). Artificial Intelligence and Biotechnology: Risks and Opportunities.

A note from Stuart: This newsletter has many readers who are members of Congress or congressional staffers. Given that Congress has been pushing back on science funding, someone should look into whether the funds are actually going out the door, or whether your constituents are getting stiffed for no apparent reason.

***

The National Science Foundation is currently making new research awards roughly 70% slower than its historical pace. The NIH is about 50% behind. Even DOE’s Office of Science has been unusually slow to get grants out the door. All of these agencies received strong appropriations from Congress for fiscal year (FY) 2026.

So where’s the money?

The past year and a half has been turbulent for US science. In FY25, the administration proposed cutting NIH by roughly 40% and NSF by more than half, froze and terminated grants at multiple agencies, cut staff across agencies, and pursued a range of policy changes that created uncertainty across the research community.1

In August 2025, against that backdrop, we commissioned professional forecasters to predict the next three years of US science funding. They were asked two questions. The first was about appropriations: how much money would Congress give NIH and NSF? Despite the turmoil, the forecasters predicted relative stability, with both agencies expected to land near their FY25 levels.

They were right. In early 2026, Congress passed bipartisan spending bills for FY26 that broadly rejected the proposed cuts. NIH received $48.7 billion, a $415 million increase over FY25, and NSF received $8.8 billion, a slight decrease from FY25 but more than double the administration’s request.

The forecasters were also asked a second question: how much money would these agencies actually spend? Here, they again predicted relative stability. But on this question, the jury is still out.

How federal money moves (or doesn’t)

For our purposes, the federal budgeting process has three key steps. First, Congress authorizes the use of federal resources for certain activities, then it appropriates specific dollar amounts to agencies, then the agencies obligate those funds for their appropriated purpose.

These numbers do not always align. It is not uncommon, for example, for Congress to authorize funding that it never actually appropriates.2 But the deviation that is rarer, and more relevant today, is a gap between appropriation and obligation. In a typical year, this gap is modest and uninteresting; agencies usually spend what they are given at a roughly predictable pace over the course of the year, and communicate to Congress the reasons for any differences between appropriations and obligations.

But the past two years have not been typical. Across the federal government, agencies have been slowing grant-making, terminating awards, and losing the staff capacity needed to process and manage spending.

In FY25, NIH spending slowed dramatically starting in February and into the spring, finally picking up in July thanks to pressure from advocates and members of Congress who pushed back against an attempt by the administration to freeze funding. NIH’s situation was particularly visible, with analysts and media outlets keeping a close eye on its spending rate. But other agencies were not so lucky. It wasn’t until the end of the fiscal year that it became clear that USDA had been struggling to make grants, and ARPA-E ended the fiscal year having obligated less than half of what it was appropriated.3

Tracking science spending

Despite strong appropriations, we’re now several months into FY26 and the same dynamics are playing out again. To ensure agencies are in a position to execute on their mission, policymakers will need to keep a close eye on spending; but information on grantmaking and obligations is fractured across budget execution reports, USAspending, and agency-specific databases.

To support this effort, I built ScienceSpending.org. This site pulls together up-to-date data on spending at five major agencies’ science functions to show whether spending is on pace relative to appropriations and historical trends.4

The site compares each agency’s current spending to spending rates in past years, and provides rates of new grantmaking, all award-making (i.e. including non-competitive continuations or modifications of previous years’ grants), and obligations (the broadest category of spending, including not only award-making, but personnel, admin, etc).

At time of writing, the NSF’s situation is the most concerning. NSF is making new awards roughly 70% slower than typical, and even its more general rates of getting money out of the door (both “all awards” and “obligations”) are almost 60% behind pace.5 In FY25 NIH was able to close a similarly large gap, in part by increasing its use of forward funding;6 NSF has less room to use that lever this year, as it already uses up-front funding for the majority of its grants.

NIH is also significantly behind pace in FY26, spending money on new grants roughly 50% slower than it typically has; it is also 29% behind on all award-making, and 16% behind on all obligations. Notably, these topline numbers hide substantial heterogeneity across institutes. The National Cancer Institute, for example, is even farther behind pace than the agency as a whole, having spent 79% less on new grants, and 43% less on grantmaking overall, than is typical at this point in the fiscal year. The National Institute of General Medical Sciences, meanwhile, is near its normal pace.7

Other agencies are in a better position. DOE’s science spending is a bit behind pace overall, with the Office of Science’s lack of grantmaking8 being balanced out by ARPA-E’s increased spending after having spent well under its appropriation in FY25. NASA is slightly behind its typical pace, and USDA is roughly on track.

There are, of course, legitimate reasons why an agency might be above or below its historical pace in any given month. Government shutdowns delay processing; new policies require revised procedures; large coordinated funding programs can shift the curve. But having insight into patterns within and across agencies can be helpful for spotting troubling or hopeful trends, and understanding when and where attention is needed.

In most years, this kind of tracking would be an esoteric exercise. I hope it will be again soon. Until then, thanks for keeping an eye on it.

***

The White House’s fiscal year 2027 budget proposal, released earlier this month, would cut the National Institutes of Health (NIH) funding by ~$5 billion. This is a meaningful cut, though far smaller than the roughly $18 billion reduction proposed for 2026, which Congress ultimately rejected in favor of a modest increase. Whether Congress holds the line again remains uncertain.

This moment, however, invites a broader question: beyond the size of the budget, are there ways NIH can use its resources more effectively?

NIH should expand its use of “pull” funding, starting with generic drug repurposing, while continuing to support its core “push” funding model. With an annual budget of roughly $40 billion, NIH is the world’s largest funder of biomedical research. Its traditional grant-based approach, often focused on areas the private sector won’t invest in, has helped drive breakthroughs including the first HIV/AIDS drugs. Given its significant role in biomedical research, even modest improvements in how NIH allocates its funding could have outsized impact.

What is “push” and “pull” funding?

Most of NIH’s budget flows through push funding such as grants awarded upfront. This model has powered decades of discoveries but requires funders to decide in advance who is best positioned to solve a problem and what approach is most likely to work.

Pull funding takes a different approach. Instead of paying upfront, it pays based on outcomes. An agency using pull funding would define a goal, such as demonstrated adoption of a drug for a new use, and offer a reward to whoever achieves it.

Push and pull funding are complements, not substitutes. Push funding is most valuable when NIH can identify strong projects using publicly available information, or in early-stage science where both public and private knowledge are limited. Pull funding is most valuable when researchers have private information that NIH cannot easily access. Since it only pays for success, pull funding filters for the most promising ideas.

In other words, push funding supports the promising opportunities NIH can already see. Pull funding helps crowdsource outside information and effort.

A useful analogy comes from the Wild West, where sheriffs pursued the most promising leads themselves but also posted bounties to draw in information and effort from outsiders.

Why focus on drug repurposing?

Generic drug repurposing represents a clear market failure, in need of public funding, and is also a natural fit for pull funding.

Why does generic drug repurposing require public funding? When a drug’s patent expires, “generic” competitors enter the market and drive prices down. With low prices and no exclusive rights, companies cannot profit enough from a new use approval to justify funding expensive clinical trials. Promising treatments go unstudied not because the science is weak, but because the business case is missing.

Figure 1. The probability of approval for new use decreases after market exclusivity expiration

Source: Budish, et al. 2025
Note: A standard patent expires after 20 years.

Why is pull funding especially well-suited for drug repurposing? The key advantage of pull funding is that it taps dispersed, private knowledge. Clinicians may spot patterns long before they are formally tested. For example, decades after aspirin was introduced, Dr. Lawrence Craven observed its effects on bleeding and suggested it could prevent heart attacks. Years later, researchers confirmed this idea, and aspirin now reduces the risk of a second heart attack by about 20 percent for millions of Americans.

Additionally, companies may hold similar untapped insights. They have unpublished clinical trial data and analyze real-world data.1 Both can provide insights into promising new uses.

Pull funding can also improve adoption. Even when research identifies new uses for generic drugs, clinicians are often slow to prescribe them. For example, metformin for prediabetes, a use identified through NIH-supported research, is used in only about 2 percent of eligible patients despite its significant benefits. Pull funding that rewards clinical impact, not just trial results, can help close this gap.

What a pilot could look like

A practical starting point is an NIH pilot that offers financial rewards for successful generic drug repurposing, with payments tied to impact and contingent on FDA approval. Eligibility should be broad, open to universities, startups, nonprofits, and established companies, since the strength of pull funding lies in attracting contributors the funder cannot identify in advance.

Pull funding could work as follows. First, the funder would commit to reward research sponsors based on the expected health impact of a repurposed drug. Second, in response, researchers would investigate new uses for generic drugs. Third, clinical trial data would then be used to assess impact. Fourth, if successful, manufacturers would bring the drug to market for the new use. Fifth and finally, NIH would reward the entity that generated the evidence (i.e., the trial sponsor), rather than the manufacturer, based on the drug’s measured value and real-world adoption.

Figure 2. Key design features of a pull funding model for generic drug repurposing

The generic drug repurposing program could further target areas with especially weak commercial incentives, such as drugs for healthcare emergencies and neglected tropical diseases. NIH already plays a central role in these disease areas through the National Institute of Allergy and Infectious Diseases.

This program may need new legislation to begin, but it’s possible that NIH could use existing authority, such as other transaction authority and/or prize authority, to run these programs. It could also coordinate with other US agencies, including the Centers for Medicare and Medicaid Services, Biomedical Advanced Research and Development Authority, and the Department of Defense. For example, as discussed in our white paper, Medicare and Medicaid’s Innovation Center could fund generic drug repurposing when it presents cost-savings. The NIH could then complement this effort by supporting high health-impact opportunities that do not yield immediate cost-savings.

Conclusion

At a time of budget pressure, it is important to consider how NIH can use its budget most effectively. Pull funding offers a practical complement to existing push funding, and drug repurposing is a natural place to start. By rewarding results rather than proposals, pull funding can surface hidden information, accelerate drug repurposing, and ensure proven treatments are actually used.

To learn more, please reach out to Sarrin Chethik: schethik@marketshapingaccelerator.org.

Few topics relevant to research produce more confusion and, in many cases, acrimony, than so-called “indirect costs”. This term, also referred to as “Facilities and Administrative Costs”, relate to research costs that are not readily attributable to any specific project. This turmoil is due to two primary issues and their consequences.

First, the estimation of indirect costs is inherently complicated and arcane. This requires considerable accounting and access to lots of institution-wide data. Second, both institutions and the government have not worked to make indirect costs even reasonably transparent. Indeed, in some cases, some organizations have gone out of their way to obscure important information.

These two factors have inevitable consequences. Many people, including most faculty who bring in federal grants as well as some governmental officials at funding agencies, do not understand indirect costs. In some situations, investigators may perceive that indirect costs represent an unfair tax on the grants that they bring into their institutions. In the political context, indirect costs are an easy target for those with other agendas regarding science funding and universities. Throughout this discussion, it is important to know that the recipients of federal grants are the institutions from which the applications arise, not the investigators.

What are indirect costs and how are they determined?

Many different costs are associated with conducting research. “Direct” costs include full or partial salary for the principal investigator and for others involved in conducting the research, costs for reagents and other supplies, and other expenses depending on the project. This first group of costs are relatively easy to assign to a specific project and the associated grants.

There are other costs of conducting research that are “indirect” and are difficult to attribute to a specific research study. These include costs of heating, cooling, and cleaning laboratory space, administrative costs related to personnel and other aspects of the research (e.g. institutional review boards, grants management/accounting, conflict of interest management), depreciation of laboratory and office space (which reflects the fact that these lose value over time due to wear and tear), animal care and use facilities, and so on. Only some small fraction of each category is related to any given project.

At some points in time and, still, in some systems in other countries, the second group is treated as direct costs with a requirement for estimating the amount in each category to assign to each project. Around the beginning of the 1960s, a more efficient process was developed. This was based on the insight that indirect costs only need to be reconciled at the institutional level, not at the level of individual grants. For each institution, an overall indirect cost rate was determined essentially by looking at all the indirect costs across the institution and dividing by the sum of the direct costs. One important adjustment is that some direct costs are excluded from indirect cost calculations. Examples include small equipment since this is generally housed in space already used for research and the equipment does not require any administrative expenses. Aside from the cost of the experimental intervention itself, many patient care costs associated with clinical trials (such as blood work or costs of treatment related to side effects from the experimental intervention) can be reasonably assumed to be covered by the hospital or health care system/insurance as part of patient care. Removing such costs results in what are called “modified total direct costs” (MTDC) which is then multiplied by the indirect cost rate to yield the indirect reimbursement due.

The principles for negotiating indirect costs for educational institutions are described in 2 CFR 220, formerly known as OMB Circular A-21. The principles for other non-profit institutions including independent hospitals and free-standing research institutes are governed by a slightly different document 2 CFR 230. Indirect costs are broken down into “facilities” and “administrative” costs. Importantly, the administrative portion of indirect costs has been capped at 26% since 1992 for educational institutions under 2 CFR 220 but is not capped for hospitals and research institutes.

Indirect cost rates are calculated across the federal government rather than agency by agency. In other words, at a given institution, the indirect cost rate is the same for a grant from the National Institutes of Health as it is for the National Science Foundation. Each institution negotiates its indirect cost rate with one of two centers within the government. For institutions that tend to do more biomedical research, rates are negotiated with an office within the Department of Health and Human Services (not the NIH) and are typically negotiated every four years. For those that tend to do more physical sciences research, rates are negotiated within the Office of Naval Research and are typically negotiated annually.

Again, once a negotiated rate is agreed to, the amount of indirect costs for each grant is calculated by taking the direct costs for the grant, removing some of the costs not subject to indirect costs to yield the MTDC, and then multiplying by the indirect cost rate.

To give a concrete example, suppose that a project has direct costs of $300,000, with $90,000 of that attributable to equipment not subject to indirect costs. Assume that the indirect cost rate is 55.0%. The indirect costs due would be 0.55(300,000 – 90,000) = 0.55(210,000) = $115,500. The total costs received would be $300,000 + $115,500 = $415,500, and the percentage of the total costs due to indirect costs would be $115,500 / $415,500 = 27.8%. These values are typical for actual indirect costs as will be discussed subsequently.

Key aspects of Indirect costs are explained well and engagingly in this short video:

.

Details about indirect costs are, unfortunately, difficult to find. Universities rarely share the components that contribute to indirect costs and that results in a continued sense that they are hiding something. However, occasionally, some universities do share some details. For example, Princeton has this information on their website, which I recently stumbled across.

Here “OR” refers to Organized Research (that is, research that is conducted in campus facilities) and “OSA” to Other Sponsored Activities (often performed off-site). This reveals the dominant categories that constitute the indirect cost rate. Unfortunately, since similar data are not readily available for other institutions, it is not possible to examine how these components vary from institution to institution. Note the capping of Administrative Costs at 26.0%.

Common misunderstandings about indirect costs

Indirect cost rates are determined by the funding agencies

Many believe that the funding agencies themselves play a role in setting indirect cost rates. As noted above, two centers (termed “cognizant agencies”, one in the Department of Health and Human Services and one in the Office of Naval Research) negotiate indirect cost rates. Each organization deals with only one of the cognizant agencies.

There should be different indirect cost rates for different grants with the first grant to an investigator receiving a higher rate, but subsequent grants receiving low rates

In terms of actual indirect costs, this may seem to make sense. It seems that the first grant to an investigator has indirect costs that should cover most lab space and administrative costs. Thus, indirect costs for the second and subsequent grants should be lower. However, this does not reflect how the indirect cost rate is determined based on research activities pooled across the institution. Alternative models with higher indirect rates for the first grant and lower rates for subsequent grants are certainly conceivable but the total amount of indirect costs going to the institution would not change. Since indirect costs are negotiated based on actual data about costs and well as projections about how much funded research will occur, this applies to all institutions regardless of scale.

Indirect costs are returned to investigators at some institutions

It is a practice at some institutions that the dean’s office “returns indirect costs to investigators” under some circumstances. This is purely an accounting device intended to incentivize seeking and winning more grant funding. Indirect costs cover (but usually not in toto) real costs related to research. Funds must be expended for research to proceed: Administrators who manage compliance issues must be paid. The notion of “returning indirect costs to the investigator” is purely accounting. Unfortunately, this has contributed to the notion that indirect costs form slush funds that can be used for any purpose. Note that some costs of departmental administration are allowable indirect costs as the Princeton example about illustrates.

Institutions make a profit on indirect costs

This appears to be a widely held view, but it is inaccurate. Indirect cost rates are negotiated based on actual expenditures. The full costs associated with research are not, in fact, covered. This is due in part due to the cap on administrative costs (put in place in 1992) while administrative expenses related to compliance have increased substantially. Indirect cost rates are negotiated based on projections of future research activities so institutions can do a bit better, in terms of better indirect cost recovery, if they beat those projections but, even in the best case, they still are unlikely to cover their full indirect costs.

The February 7^th, 2025 NIH indirect cost rate notice

On February 7^th, 2025, a notice was posted by NIH (although it was not written nor vetted by anyone at NIH) that asserted that indirect cost rates would be capped at 15%. This notice caused great concern across the academic community. The approach described in this notice has been challenged and found to be illegal, including in a recent appellate court ruling I will discuss subsequently. This notice also is based on some misstatements, perhaps made from ignorance. Here is one key passage:

Although cognizant that grant recipients, particularly “new or inexperienced organizations,” use grant funds to cover indirect costs like overhead, see 89 FR 30046–30093, NIH is obligated to carefully steward grant awards to ensure taxpayer dollars are used in ways that benefit the American people and improve their quality of life. Indirect costs are, by their very nature, “not readily assignable to the cost objectives specifically benefitted” and are therefore difficult for NIH to oversee. See Grants Policy Statement at I-20. Yet the average indirect cost rate reported by NIH has averaged between 27% and 28% over time. And many organizations are much higher—charging indirect rates of over 50% and in some cases over 60%.

This statement said that “the average indirect cost rate reported by NIH has averaged between 27% and 28% over time”. It is true that indirect cost recovery (indirect costs paid, divided by total costs paid) have averaged 27-28%. But these are not indirect cost rates. This difference is due to (1) the fact that some direct costs are excluded from indirect cost calculations (as noted in the discussion of MTDCs above) and (2) Indirect costs are determined by multiplying the indirect cost rate by the MTDC whereas the 27-28% figure is the percentage of total costs. Thus, if the indirect cost rate is 50% and all direct costs were included, the percent expenditure would be 0.50 (DC) / (DC + 0.50 (DC)) = 33.3%. Thus, this statement is an apples vs. hummingbirds comparison which serves to create the false impression that many organizations are extracting substantially more indirect cost reimbursement than they are due.

Legal Issues

As noted above, the 15% cap was not implemented because organizations, including the American Association of Medical Colleges, quickly sued claiming that the proposed policy was illegal. The lawsuits were successful and have been upheld on appeal. The primary reason is that Congress established the indirect cost rate negotiation process and routinely incorporates language about this process in appropriation laws. Thus, any changes to the policy would have to developed by Congress and passed into law.

Negotiated Indirect Cost Rates and Indirect Cost Recovery

Negotiated indirect costs rates are necessary for applicants to complete grant applications, and they can generally be found on organization websites through (sometimes persistent) internet searches. Unfortunately, they are not compiled in any central location, and, in some cases, the information is password protected or only available from inside institutional networks.

However, I discovered that they can be readily deduced from data publicly available from NIH Reporter. For institutions with a reasonable number of grants (50 or more) in a given year, a scatter plot of the Indirect_Cost_Amount versus the Direct_Cost_Amount reveals points that almost all lie on or below a limiting line. The slope of this line is the indirect cost rate.

Data

The points that lie on the line are those where all direct costs are subject to indirect costs. The points that lie below the line are those where the modified total direct costs (MTDCs) are less than the total direct costs.

Using this approach (with confirmation from internet searches), I compiled the negotiated indirect cost rates for the top 100 institutions in terms of annual NIH funding for fiscal year 2024. I also calculated the indirect cost recovery percentages for these institutions (total indirect costs) / (total direct costs + total indirect costs). The results are shown below and in the file (link):

Estimated Indirect Rates for Fiscal Year 2024 derived from NIH Reporter Data

(Sorted based on decreasing grant funding received)

The distribution of these negotiated indirect cost rates is shown below:

This histogram reveals that these rates range from 43.0% to 96.9%.

However, examination of actual indirect cost recoveries reveals a somewhat different picture:

The distribution is much tighter, with a range with a range of 21.8% to 37.0%, median of 28.9%, mean of 29.1% and standard deviation of 3.0.

This analysis reveals the basis for the statements in the February 7^th notice. Negotiated indirect cost rates do vary substantially from institution to institution, but the actual indirect cost recovery is much lower and much more tightly clustered.

The top 10 institutions with the highest negotiated indirect cost rates are:

These are all free-standing hospitals or independent research institutes, not subject to the 26% cap on administrative cost reimbursement. Despite these high negotiated rates, the actual indirect cost recoveries range from 30.4% to 37.0%, only slightly higher than from institutions with lower negotiated rates.

Alternative Indirect Cost Reimbursement Plans

Because of the controversy around indirect costs, alternative models for accounting and reimbursement are under active development. One of these is the so-called FAIR (Financial Accountability in Research) model. The primary goal of this model is to have indirect cost reimbursement be justified on a grant-by-grant basis rather than at an institutional level, at the expense of substantial additional accounting required by applicants. It is not clear whether this model will decrease, increase, or keep constant actual indirect cost reimbursements.

The NIH Director has also made comments that “market forces” might be used to allow institutions to compete based on lower indirect cost rates (which would then be taken account in making funding decisions); few details have been provided. It is not clear if institutions might choose to appear more competitive by underbidding their actual indirect costs. Since these costs are real, increasing the institutional level of federal funding while losing money on every new grant does not seem to be a robust business model.

In any event, Congress would have to act to approve any model that differs from the current model. It is unclear how these issues will surface during the current appropriation processes.

Policy Recommendation Principles

In considering possible modifications to indirect cost policies, several principles seem important:

First, indirect costs are real research costs. One cannot do research in a sustainable way with laboratory space that is maintained, access to journals, appropriate administrative support for animal and human subject protection, and so on. Those costs could be transferred from indirect to direct costs but with no cost savings for anyone and, likely, increased administrative burden.

Second, increase transparency for all parties would be beneficial for generating support understanding and support. I would favor institutions being required on public websites to post their indirect cost rate information in a manner at least as detailed as that posted by Princeton shown previously. Moreover, a website should be developed with indirect cost rates posted, updated whenever they are renegotiated. This would not require extensive infrastructure to achieve if all institutions were compelled to cooperate. These are, after all, taxpayer funds that are being committed.

With those principles in mind, I am agnostic about the current model versus the FAIR model (which, again, does away with indirect cost rates pe se, but required indirect costs to be estimated for each grant. It may not be possible to sustain the current model due to the damage done through decades of lack of transparency although, in my view, it was a successful early effort aimed at governmental efficiency. The FAIR model has the advantage that the indirect costs for each grant could be examined and audited. However, this would come at the expense of increased administrative burden that could fall on investigators, depending on how the FAIR model is implemented at each institution.

While there are few detailed about the “market forces” plan, if I am correct that the plan would allow institutions to propose indirect cost rates that they would be willing to accept, then I have substantial concerns. First, indirect costs are real research costs that have to come from somewhere and institutions already have some pressure to manage these costs effectively (since indirect cost reimbursements do not fully cover indirect costs). Thus, the need to compete based on lower indirect cost rates could encourage lower levels of building maintenance, decreases in library offerings, cutting administrative activities related to animal care and patient safety, and so on. This would not benefit research in the long run. In addition, it is unclear if this would benefit institutions that currently have lower indirect cost rates as some institutions with substantial other funding to support their research activities might be able to afford relatively low bids.

Let me close with a comment about the rationale for the February 7^th NIH notice. This notice stated:

Most private foundations that fund research provide substantially lower indirect costs than the federal government, and universities readily accept grants from these foundations. For example, a recent study found that the most common rate of indirect rate reimbursement by foundations was 0%, meaning many foundations do not fund indirect costs whatsoever. In addition, many of the nation’s largest funders of research—such as the Bill and Melinda Gates Foundation—have a maximum indirect rate of 15%. And in the case of the Gates Foundation, the maximum indirect costs rate is 10% for institutions of higher education.

There are two problems with this argument, one factual and one numerical. As a matter of fact, not all universities and research institutions accept foundation grants that provide low levels of indirect cost reimbursements. A small number of institutions choose to forgo research opportunities for their investigators instead if they do not come with something close to full negotiated indirect cost rates.

The numerical argument relates to scale. Suppose that you are at an institution that is approximately 50^th in NIH funding with has $170 million of total cost in federal funding, an indirect cost rate of 56.0%, and an indirect cost recovery of 28.0%. In addition, your institution receives $10 million in direct costs of foundation funding with an indirect cost rate of 15%.

The total amount of indirect costs received would be (0.28 * $170 million) + (0.15 * $10 million) = $47.6 million + $1.5 million = $49.1 million

Declining to accept the $10 million in foundation funding would reduce the total research budget by $11.5 million (out of $181.5 million) and indirect cost recovery from $49.1 million to $47.6 million and would mean the loss of potentially important research projects.

Now supposed that the federal indirect cost rate was unilaterally reduced to 15%. Further assume that this rate applies to all direct cost without modification, the most favorable (for institutions) interpretation of the ambiguous February 7^th notice. I will not go through the algebra, but this would mean that the federal indirect costs recovered would be reduced to $18.36 million, a loss of $29.24 million. If, instead, the indirect cost rate applied only to the modified total direct costs, the indirect costs recovered would be reduced further to $12.75 million and a total loss of $34.85 million. These represent very substantial reductions out of a total research budget of $170 to $181.5 million.

This calculation reveals why institutions acted so quickly and aggressively to bring litigation to stop implementation of the 15% indirect cost cap, namely that this would have led to 15-20% reductions in total research budgets with real cost gaps to make up with non-federal funds. As well, they brought litigation quickly because the unilateral change in indirect cost policy without the involvement of Congress was clearly illegal.

A RESPONSE FROM DR. BERG:

I read with interest “Venture Capital Has Lessons for Government and Philanthropy” by Khanduja and Buck. Discussions and analyses of the benefits and weaknesses of different approaches to providing science funding are important, but these should be based on well-informed perspectives and not on anecdotes and straw man arguments. I will focus of funding by the National Institutes of Health (NIH) as I have decades of experience as an applicant, a grantee, a peer reviewer and study section chair, an NIH institute director (the National Institute of General Medical Sciences, NIGMS, 2003-2011), and a trans-NIH program development leader. I will illustrate my observations with specific examples, all related to discoveries related to DNA and RNA biochemistry and its applications.

There are many misunderstandings and myths in this piece. Perhaps the most fundamental one is captured here.

By the time you get the funding (if at all) and execute the research plan, you might learn that your first idea wasn’t quite right and that an even better approach might work (*Marcia McNutt told one of us that this is what regularly occurred when she was a practicing scientist). But if you try to amend the grant, you will need to navigate the federal bureaucracy once again.

This fundamentally misunderstands that most NIH funding comes in the form of grants, not contracts. My experience and that of essentially every practicing scientist I know is completely aligned with that ascribed to Dr. McNutt. In the course of a executing a research plan, you discover the need for alternative approaches or learn of results, presented or published, that affect your chosen research topic. But you do not have amend anything! You pursue the science and describe whatever changes in approach in your annual progress report. NIH program officers understand that this is the nature of the research and, indeed, often that your are working in a desirably dynamic field.

An often-discussed example of this is represented by the work of Tom Cech when he was a young investigator at the University of Colorado. His proposal was to purify and characterize the protein enzyme responsible for splicing a particular RNA molecule, removing internal portions that were not present in the mature, functional RNA molecule. During these studies, Cech and his coworkers discovered that the RNA would splice itself, without the need for any protein at all! He was able to pursue this discovery, focusing on the requirements for this RNA-driven process, without contacting NIH or amending his grant in any way. When he submitted his grant renewal and reported that no protein was required, the study section response (according to Dr. Cech) was “Cool. That’s even better.”

Several paragraphs later, the piece continues with this:

Program officers face perverse incentives too. If a major federal initiative funds a bunch of failed ideas, Congress might hold a hearing. But if a given grant succeeds brilliantly, Congress doesn’t hold a ceremony to give the program officer a medal. Agency bureaucrats get paid the same salary either way, with no upside for identifying breakthroughs and serious downside for visible failures.

In my view, this completely misunderstands or misrepresents the motivation of program officers and the relationship, cultivated over decades, between the NIH and Congress. Program officers are public servants who are not motivated much by money but rather by helping science advance. As an institute director, I oversaw the administration of the small bonuses and discretionary salary increases available to program directors in my institute. We tried to reward hard work and creativity, providing good guidance to their applicants and grantees, and teamwork within the institute and in cross-institute initiatives. We never even considered giving any special compensation to a program officer whose grantee received a Nobel Prize (and there was at least one Nobel Prize awarded to an NIGMS grantee in 6 out of the 8 years I was director). The program officers across the institute were already proud enough of having contributed in some way to a discovery that was so recognized. NIH directors, as well as individual NIH institute and center directors, have spent many decades informing members of Congress and staff about how NIH funding works. I know of no examples where there have been Congressional hearings about funding “failed ideas”. Congress may express concern or frustration over slow progress on an important health issue, but they generally understand that this is almost always due to fact that science, in general, and biology in human populations, in particular, is hard.

The section concludes with a discussion of Katalin Karikó and her role in developing mRNA vaccines. It is true that she struggled to get NIH funding for this work, likely because it was perceived as too risky. I will discuss risk tolerance at NIH below. I find her university’s decision to demote her because of these funding challenges more troubling as it represents what I see as a more serious flaw in the biomedical research system, particularly at medical schools. What is missing from the discussion in the essay is the timeline, Karikó’s demotion was in 1995 and the DARPA funding noted occurred in the early 2010’s. Thus, more than 15 years were required to move her ideas forward, even with support from other sources including BioNTech. Much remained to be discovered to enable mRNA vaccines including development of an understanding of the innate immune response to single-stranded RNA. Perhaps the major enabler of mRNA technology was the discovery that modified bases could be introduced that would not trigger the innate immune anti-viral responses but would still act like natural bases for protein coding. Effective Liposome-based deliver systems also were needed, These discoveries were not made until the early 2000s, in studies unconnected to potential mRNA technologies. I doubt many venture capital-style funders would have the patience necessary for these long and uncertain timelines.

I now return to the conservative nature of NIH peer review and funding. In my view, there are two major factors that contribute to this. First, being a successful researcher generally requires identifying many of the potential problems associated with experiments. Because of this, many, but not all, scientists are relatively conservative, readily identifying and articulating potential challenges. I witnessed an example of this when I was NIGMS Director. Mario Capecchi shared the Nobel Prize in Physiology or Medicine in 2007. He was a long-time NIGMS grantee. I was shocked when I saw him quoted saying the NIH had not funded his Nobel-winning research on homologous recombination used for gene modification because I knew that NIGMS had funded him continually for many years. It turned out that he was referring to a comment in his NIH peer review critique that this part of his proposal was “not worthy of pursuit” as it was unlikely to work. But he still got a very good score overall and NIGMS fully funded his proposal. This again illustrates that power of NIH grants as opposed to contracts. He was free to pursue this approach and he did so, getting it to work somewhat in about four years.

More important, in my view, is the fact that NIH study sections receive a very large number of high-quality applications. Empirically, it seems that most reviewers will favor, at least somewhat, high-quality proposals that are almost certain to yield interesting and important, even if incremental, results over proposals that have potential but might yield little. However, there is a straightforward solution to this which has been implemented at NIH for decades, namely having programs that are specifically for high risk-high reward research.

Applications are submitted that are supposed to contain high risk-high reward ideas and reviewers are specifically selected and instructed to look for such projects. If a proposal is solid and well-supported by preliminary data, then it should not be a priority for funding through such programs. On the other hand, a proposal that addresses an important problem and it appears that the investigator(s) has a novel approach is ideal for such a program. I helped run the NIH Director’s Pioneer Program for 2005 through 2011 and helped create the partner NIH Director’s New Innovator Program beginning in 2006, while I was at NIH. These programs are still in place. They might need tweaking and expansion, but they were thoroughly evaluated in the past (https://commonfund.nih.gov/pioneer/programevaluation and https://commonfund.nih.gov/newinnovator/programevaluation) and were deemed to be successful in achieving their main goals (compared to other NIH funding programs).

The question for NIH leadership is much like that for a financial portfolio manager. What percentage should one invest in solid, dividend-paying established companies and what percentage should be devoted to start-ups that might flame out but might lead to the “next big thing”? This can be managed easily by setting aside funding for these different programs without the need for radical experiments that have not already been tested.

In summary, I agree that funding approaches at agencies like NIH should be examined thoughtfully and creatively. But this should be done will full knowledge of how these agencies actually function. As is almost always the case in medicine, it is essential that one arrive at the right diagnosis before deciding on a treatment.

Foundation and Venture Capital Funding

Although I have less experience with foundation and VC funding than I do with federal grants, I do have some experience. When I became a department director at Johns Hopkins University in 1990, the rebuilding of the department was being supported, in part, by the Lucille P. Markey Charitable Trust. This foundation has a substantial impact on several fields of biomedical research.

However, two points deserved comment. First, the areas of impact were limited because of the limitations on the resources available, both financial and staff expertise. The trust chose a small number of specific areas in which to specialize and hired program officers to cover those areas. This is, in my experience, typical of most foundations and this limits to ability of foundations to cover a wide range or emerging areas of science. Second, many foundations are limited by the resources that they can make available as they general spend only some or all the income generated by their endowments. The Markey Trust had a relatively large impact because they spent down all their resources including the principal over a 15-year period. Foundations can provide flexible funding that can be used for high risk-high reward research, but foundation support is often used for infrastructure that can be hard to support through other mechanisms.

Although I have never founded a company, I have served in the past on the scientific advisory boards of three venture capital supported companies beginning at the time of their founding. The first company was a structure-based drug design company called Three-Dimensional Pharmaceuticals. The company hired some talented scientists and developed some impressive technology.

However, my main impression was that the venture capitalists involved waited until the company was on the verge of financial failure before stepping in, obtaining larger and larger equity in the company, and replacing the very creative CEO who had built to company with leadership with the goal of selling to company without much emphasis on new technology development. This did occur and the company was sold to Johnson and Johnson.

The second company was Sangamo Biosciences which was originally focused on zinc finger proteins, a group of gene regulatory proteins that I had played a major role in discovering and characterizing. The company was founded by a relatively savvy CEO with industrial experience. Their strategy was largely based around wide control on intellectual property. Sangamo did develop some impressive technology and applications and it still an independent, publicly traded company.

The third company was Gryphon Biosciences which was founded to investigate proteins made out of D- rather than L-amino acids. The company dissolved within a year due to lack of financing. I note the this field is still an active area of study with several companies working in this space more than 30 years later although there are no huge practical applications that have gone to market to date.

While this is a limited experience set, at no time in these interactions did I see any evidence that venture capital approaches were particularly effective in driving scientific advances. The venture capital approach was useful to help guide technologies to potential markets and partners which is something with which most academics including myself have limited experience.

My experience was that the venture capital approach was to hear lots of ideas and pick a small number on which to focus, based on their potential marketability in the short or, at best, the medium term. Many ideas that had had more potential in the long term to answer important questions and needs in the long term were discarded.

A REPLY FROM AISHWARYA KHANDUJA AND STUART BUCK:

We’d like to thank Dr. Berg for his gracious response (and his continuation of the meme!), and would just like to add a few notes in reply.

As an initial matter, it looks like we are all basically agreed on a significant number of points from the original essay. That is, Dr. Berg doesn’t contest the points (most of which are not original to us!) that:

• traditional government funding does require too much paperwork

• reporting requirements as to time and effort can be overly burdensome

• most traditional funders take too long to get money out the door

• being a good science funder requires a different skillset from that for being a good scientist

• the government doesn’t have enough focus on funding so-called “high risk, high reward” research outside of a handful of small programs

• both government and philanthropy are less subject to market competition than VCs

• NIH reviewers favor “proposals that are almost certain to yield interesting and important, even if incremental, results over proposals that have potential but might yield little”

• Katalin Karikó was mistreated by her university

• government efforts to fund small businesses are often run by people with no relevant experience as a VC funder.

So, to focus on the few areas where Dr. Berg identifies a strong disagreement:

First, Dr. Berg points out that the burden of NIH bureaucracy is less than we imagined, because there is actually no need to amend a grant if you “discover the need for alternative approaches or learn of results . . . that affect your chosen research topic.”

At the same time, the official NIH Grants Policy Statement says that a grant recipient “must obtain prior approval [emphasis in original] from the NIH” for a change in a grant’s scope, with the first example being a “change in the specific aims approved at the time of the award.” Other examples of prior NIH approval needed (there are many more): “substitution of one animal model for another,” “shift of the research emphasis from one disease area to another,” “changing assays from those approved to a different type of assay,” or “purchase of a unit of equipment exceeding $25,000.”

It seems obvious to us that this policy is much more restrictive than any VC funder, regardless of the details as to which actual activities need advance approval or not. Our point here is about the relative balance of bureaucracy vs. flexibility/discovery, not about which precise activity lands in the category of “NIH will allow an exception.” Because NIH and other federal science-funding agencies are funded with taxpayer monies and are subject to sometimes irrational Congressional oversight, federal agencies tend to be quite bureaucratic and not as flexible and agile as many private funders including VCs.

A second major disagreement is over our assertion that program officers (and the like) ought to see some financial reward for being prescient enough to fund scientific work that would turn out to be groundbreaking. Dr. Berg objects on the ground that government funders are “not motivated much by money but rather by helping science advance,” and by personal pride in “having contributed in some way to a discovery.”

We agree that today’s public servants are likely motivated by those rationales, not by money (or else they would be doing something else with their careers). All the same, we are living in a world where, outside of academia, smart people with expertise in science face a dichotomous choice: government service with lots of job security (at least in most typical years) and little chance for financial upside, or private careers in finance, tech, etc., with no job security but lots of potential upside.

We suspect that government might be able to recruit from a larger talent pool if there was more room for a brilliant decision to result in financial upside. We also suspect that government and philanthropy program officers might be more risk-tolerant if there was more potential upside for themselves. [And we could systematically test this sort of question via prospective metascience interventions at the policy level!]

We might well be wrong, but we can agree to disagree.

Relatedly, Dr. Berg notes, “I know of no examples where there have been Congressional hearings about funding ‘failed ideas.’” Here, we admit to a slight overstatement based on the past 50 years of everything from Senator Proxmire’s “Golden Fleece” awards to the recent DOGE attempt to ridicule seemingly-silly scientific studies. Our only point is that the public scrutiny here can be highly asymmetrical.

Third, Dr. Berg concedes the “conservative nature of NIH peer review and funding,” but notes that “there is a straightforward solution” that has already been implemented: specific programs that target so-called “high risk, high reward” research.

We agree. Indeed, we applaud programs like the NIH Director’s Pioneer Program and the New Innovator Program. But those programs are fairly small in the grand scheme of things. The Pioneer Program typically funds all of eighteen people per year, while the New Innovator Program isn’t much bigger (56 people per year). By comparison, the NIH makes about 9,000 grants per year. We think it is still fair game to critique the other 98% of NIH grantmaking.

Most basic science in the US is funded either by government or philanthropy, which collectively donate over $100 billion every year. Traditional government funding offers three things private investors usually can’t match.

• Democratic accountability: agencies like NIH and NSF have deployed billions of dollars guided by public interest rather than profit.

• Massive scale: government has created scientific infrastructure that required collective action at scale, such as particle accelerators, the Hubble Space Telescope, and national laboratories.

• Long-term thinking: Agencies maintain decades-long research programs whose benefits emerge only over generations, such as (D)ARPA funding the R&D in the 1960s that eventually became the Internet.

Philanthropy has its pluses as well. The Gates Foundation could pour billions into malaria research because of the effect on global health, not because it would generate returns.

Government and Philanthropy Have Serious Structural Rot

Despite their historical successes, both traditional government and philanthropic funding have developed structural pathologies that actively hold back scientific progress. We think they should do more to imitate the VC model of funding that has enabled the startup scene in Silicon Valley and elsewhere.1

Government’s Bureaucratic Nightmare

To paint a picture of what traditional government funding looks like: imagine that you are a brilliant researcher with an idea that could transform your field, if it works. You spend six months writing a grant proposal that can stretch to over 100 pages in total, in which you need to predict the next five years, specify your methods in detail, and provide preliminary data proving your idea works (even though you need the grant money to generate that data in the first place).

You submit this proposal to NIH or NSF and wait for many months while your proposal is reviewed by scientists who are often your intellectual competitors. As is well-documented, review committees tend to favor safe, incremental projects over truly new ideas that might fail.

By the time you get the funding (if at all) and execute the research plan, you might learn that your first idea wasn’t quite right and that an even better approach might work (*Marcia McNutt told one of us that this is what regularly occurred when she was a practicing scientist). But if you try to amend the grant, you will need to navigate the federal bureaucracy once again.

Imagine if entrepreneurs—from laundromat owners to startups—had to go to the Federal Reserve to get financing for their business. Our economy would be at a standstill if that was the process. But that is how we treat science. With such centralized funding opportunities in science, it’s no wonder the process is suboptimal.2

Program officers face perverse incentives too. If a major federal initiative funds a bunch of failed ideas, Congress might hold a hearing. But if a given grant succeeds brilliantly, Congress doesn’t hold a ceremony to give the program officer a medal. Agency bureaucrats get paid the same salary either way, with no upside for identifying breakthroughs and serious downside for visible failures.

Consider what happened with mRNA vaccine research. Scientists like Katalin Karikó spent decades struggling to get NIH funding for mRNA research. Grant reviewers thought it was too risky. Karikó’s university demoted her more than once, ultimately driving her to move to BioNTech, where she had the freedom to pursue the work that ultimately enabled the COVID vaccines. DARPA did fund mRNA vaccine research in the early 2010s, but the then-director Arati Prahabhakar told one of us that people at NIH said she was crazy for sponsoring such research.3

Philanthropy Is Little Better

Donor preferences can be arbitrary and fashion-driven. For example, a tech billionaire might get interested in longevity research, and therefore that field gets overfunded while equally important areas languish. When a foundation’s priorities shift because they hired a new President with different interests, valuable research programs get orphaned. Moreover, many foundations have become bureaucracies that mirror the government’s worst features: credentialism, reporting requirements, etc.

Another problem is that of coordinating multiple funders. Imagine you’re a researcher who needs $2 million to complete an important project. You get a $500K grant from Foundation A, $300K from Foundation B, and $400K from Foundation C, and the rest of the money from NIH. Now you have to spend a huge amount of time managing four different sets of reporting requirements in which you pretend to have spent each pot of (completely fungible!) money on a very specific list of tasks, and navigating four different program officers’ expectations. Each funder wants credit for your success but none wants to fully fund the work.

Indeed, the reporting requirements from traditional funders are often as nonsensical as going to Best Buy to get a new laptop, and then demanding to see a receipt stating that your money was spent on the letters A-M on the keyboard, on the RAM but not the internal processor, and also that Best Buy was only allowed to spend 10% of the overall bill on rent and utilities for the store.

That would be unworkable. When you buy something from Best Buy or anywhere else, you just want to know if the overall product or experience is worth the cost, and you couldn’t care less how the seller allocated all of its internal expenditures to rent, employee hours, materials, marketing, electricity, etc. Traditional funders should start caring about the ultimate results (and only that) from their grants, and stop requiring such nonsensical intermediate reporting from their grantees.

Why Venture Capital’s Structure Works

Betting on People Over Proposals

When a VC funds a scientist-turned-entrepreneur, they’re betting on that person’s ability to figure things out, not on the correctness of a Soviet-style five-year plan.

When Ginkgo Bioworks originally pitched VCs on building an “organism foundry” to engineer microbes, they didn’t have a detailed five-year plan for exactly which organisms they’d engineer and which applications would succeed. Their investors bet on the team’s ability to navigate toward valuable applications.

Over the years, Ginkgo has pivoted multiple times. Early on they focused on manufacturing specialty chemicals. Then they shifted toward fragrance and flavor compounds. They’ve worked on probiotics, therapeutics, COVID testing, detection of biological engineering, and most recently, a partnership with OpenAI to do autonomous lab experiments.

The typical NIH grant would have locked them into whatever they proposed initially, but venture capital allowed them the flexibility to change as needed. In no event would VC firms demand to see a year 5 report showing how a company’s officials spent thousands of hours of time on the exact same activities they had predicted in year 1.

We need the same attitude in traditional science funding. As former NIH official Mike Lauer recently said, “the very nature of science is that you don’t know what you’re going to be doing over the next year, for sure not five years.”

Speed vs. Bureaucracy

In early 2020, Moderna identified the Covid virus sequence and designed an mRNA vaccine candidate in 48 hours. They were in clinical trials within weeks, and had a working vaccine in under a year.

This happened not just because VC and private companies were involved, but because the government component (Operation Warp Speed) took place entirely outside the normal NIH mechanisms that would have taken the first year just to do peer review and start sending the initial checks.

Similarly, in 2021, Benchling (lab software for biotech) raised additional venture capital within two weeks. A government-funded research project would have waited for the next grant cycle, which might not come for another year. By the time additional funding arrived, the moment would have passed.

There have been occasional efforts in government and philanthropy to fund scientists quickly with minimal bureaucracy, such as Fast Grants and NSF’s rapid response grants for COVID research. But those occasions are far too rare. Both government and philanthropy need more of these flexible and speedy mechanisms.

Skin in the Game Can Create Better Incentives

Venture capitalists invest their own money and their firm’s capital. When a16z’s Marc Andreessen and Ben Horowitz backed Coinbase at a $100 million valuation, they were making a personal bet. If Coinbase failed, they lost their investment. When it went public with nearly a $100 billion valuation, they made massive returns that benefited their firm.

By contrast, government program officers invest taxpayer money. If their funded projects fail, they lose nothing personally except the possibility of public criticism. But if their funded projects succeed brilliantly, they get nothing beyond their salary. This is true even in the case of SBIR (small business) grants that NIH makes every year: neither NIH nor the personnel involved have any equity stake in any of the early-stage companies they fund. Perhaps unsurprisingly, NIH has been criticized for allowing people with no expertise in biotech or industry to make most of the funding decisions, and for having “little idea whether the SBIR program is efficient for the institution.”

For traditional science funding, the incentives are clear: avoid criticism, don’t take chances, and err towards funding established researchers at prestigious institutions doing incremental work.

Both philanthropy and government need to think about creating a set of incentives for program officers to personally capture some of the upside if a scientific grant results in a huge breakthrough. In some cases, that might be a small personal equity stake in a company, but in many cases, that breakthrough might not be a commercial company at all. In those cases, we should think about offering large financial awards, e.g., “This program officer funded early CRISPR research in 2005, and in honor of her foresight, she will receive a $1 million bonus.”

Moreover, being a good scientist is a distinct skillset from being a good science funder.4 Traditional science funding almost universally puts scientists in charge of distributing funding, whether through peer review panels or program officer stints at the NSF. We do almost nothing to identify, promote, or celebrate the skills and careers of science funders.

In VC-world, by contrast, investors and entrepreneurs occupy quite different roles. Entrepreneurs can become successful investors, of course, but the two categories still have separate incentives, identities, and career trajectories. Famously, journalists (such as Mike Moritz) and people with other backgrounds have thrived in the investor role.

We wouldn’t expect that top athletes and top coaches are always going to be the same people. Occasionally those two groups overlap, but some of the best athletes made mediocre coaches, while some of the best coaches were never superstar athletes themselves. Whether in athletics or VC, we know that performing is different from funding or coaching. Traditional science funders need to recognize this basic principle.

Risk Tolerance: The Asymmetry of Success

Benchmark led an early $11m round in Uber, and by the time of the IPO, its stake became worth over $7 billion. That one investment returned their entire fund multiple times over. The fact that they also invested in companies that failed completely doesn’t matter: one massive success compensates for many failures.

This is how venture capital is supposed to work. VCs expect 70% of investments to return nothing, 20% to return modest multiples, and 10% to return enough to make the fund profitable. Peter Thiel’s $500,000 investment in Facebook for 10.2% of the company became worth over $1 billion when he sold his shares (and would be worth far more today). The asymmetry is the point.

The federal government has trouble operating this way. Imagine a congressional hearing where it comes out that 80% of NIH-funded projects produced nothing. That would be considered a political scandal, and NIH officials would be brought before congressional committees to explain why they were responsible for so much government waste.

The public saw this firsthand when the company Solyndra failed. The Department of Energy had made a $535 million loan guarantee to this solar panel company. When Solyndra went bankrupt, it became a massive political scandal. Even though the DOE’s overall clean energy loan portfolio actually returned a profit to taxpayers, one visible failure dominated the narrative.

Philanthropic funders face similar pressure. Foundations want to tell compelling success stories. Boards want to see impact. A portfolio where most grants “fail” feels like poor stewardship, even if a few successes are transformative.

We need a change in political culture here. Members of Congress should stop highlighting individual scientific grants that they think are ridiculous or that failed. To the contrary, they should start demanding that half of all NIH projects “fail”! After all, if successful outcomes can be predicted in advance for all NIH projects, then why bother to do the research in the first place? We already know the answer! A 100% success rate means that the government is only funding marginal projects, or that the results are being exaggerated, or both.

Networks and Support Beyond Money

When Stripe started out, the YC ecosystem provided not just startup capital, but constant advice, connections to successful founders who had solved similar problems, access to potential customers and partners, help with recruiting, and a network of many alumni. Collison said this:

“Sometimes the issues are actually huge, that you are trying to close this round, this big deal, some recalcitrant investor, this stressful thing happening or it’s I don’t know how to talk to this customer, or whatever it is. I really kind of spans the gamut. I think Paul and the other partners are really good at getting that. They have done companies and they have seen this happen literally thousands of times in other companies and so whether the issue is closing this round, doing this big deal, this acquisition or whatever. Or it’s just well, how should the signup button work? They have pretty good advice. That has made a big difference to Stripe.”

By contrast, government and philanthropic grants usually provide little support other than money. We have no equivalent of the YC ecosystem in government-funded or philanthropy-funded science. We could do better.

Learning from Failure Quickly

The venture ecosystem has developed more systematic mechanisms for capturing and sharing knowledge. Industry conferences bring investors together to share their lessons learned, LP meetings create accountability for articulating what worked and why, and successful VCs increasingly write publicly about their successes and failures.

Even if much of this public knowledge is stagecraft in some sense, it is still more transparent than the government and philanthropic programs that operate on decades-long feedback cycles. Program evaluations happen years after decisions were made, by which point the people who made those decisions have moved to different roles and the context has changed so much that lessons are hard to extract.

We need more rapid feedback cycles and evaluation with regard to traditional science funding.

Venture Capital Has Real Problems

We are not claiming venture capital is perfect. It has serious structural issues that limit what it can support.

Venture capital inherently needs financial returns. This excludes important work with pure knowledge value or public good characteristics. Moreover, given the risk/reward tradeoff, the potential financial returns need to be quite high. In short, the need for unicorn-scale outcomes means VCs must reject many innovative ideas that might be valuable but won’t plausibly reach a billion-dollar scale.

Moreover, capital has grown more and more concentrated. The top 30 firms raised $49 billion recently versus $9.1 billion for 188 emerging firms. This concentration could create a bias toward established networks rather than towards outsiders like Katalin Kariko without elite credentials or warm introductions to brand-name VCs. New fund managers face pressure to raise $30 million or more, creating barriers to entry for talented but unproven investors.

Conclusion

Government and philanthropic funding remains essential for large-scale infrastructure, basic research without commercial paths, and pure public goods. Moreover, there are many reasons to think that we are creating a new category of R&D institutions that will replace much of traditional VC at some point.

But there are many lessons to learn from VC’s style and approach.

First, even imperfect incentives beat no incentives, let alone perverse incentives. The VC requirement for financial returns is often better than government and philanthropic structures that create zero personal incentive for identifying breakthroughs, and instead create strong cultural incentives to avoid visible failures. Traditional science funding should do more to create the right incentives for finding breakthroughs.

Second, and relatedly, a commercialization requirement imposes some productive discipline, in that venture-backed researchers must consider whether their work creates value for actual users. Traditional science funding should, at least in some fields, do more to sponsor scientific research that is connected to real-world firms (which was arguably the basis for Bell Labs’ success).

Third, the competitive nature of VC creates evolutionary pressure, in that firms that don’t adapt will eventually disappear as they are outcompeted by newer firms. By contrast, government funding evolves glacially, bound by appropriations processes and political constraints, while philanthropic funding evolves mainly through leadership changes that bring arbitrary changes in priorities. It’s admittedly difficult to re-create a competitive market in government or philanthropy, but we need more innovative thinking as to how to let old institutions die off and be replaced by newer and more successful ones.

Fourth, the heterogeneity of venture capital creates multiple pathways for researchers to find support. Different firms pursue different strategies, focus on different stages and sectors, etc. If one firm’s thesis doesn’t match your opportunity, another firm might be open to it.

Government programs have much less diversity as to ideas, structure, or approach. If you’re a biomedical researcher, for example, NIH is by far the biggest funder in the world. There are relatively few competitive options for finding any funding. Similarly, philanthropic funding may look diverse at a surface level, but is often fairly homogenous, as shown in the essay “The Case for Crazy Philanthropy.” We need more diversity at the institutional level. For example, imagine breaking up NIH into 10 separate organizations that compete with each other for who can find the most successful way to find cures for cancer and Alzheimer’s.

Fifth, NIH in particular has been criticized by the National Academies for running its small business program (SBIR) as if it’s a traditional research program, while not involving anyone who has biotech or industry experience. Moreover, NIH’s timeline and Phase I award process—which can result in an award of an average $327,388 after a process that “averages about 9 months” (p. 77)—is too cheap and slow to be of much use. Here is a place where NIH should literally hire people with actual VC and/or biotech experience, either as program directors or as entrepreneurs-in-residence, and let them completely overall the SBIR program.

In short, government and philanthropy should entertain bold reforms to bring VC ideas into their own work. Even when it comes to basic science, both government and philanthropy could come up hybrid approaches that mimic VC’s core advantages (backing people over proposals, speed, accountability through results, risk tolerance, skin in the game, and mechanisms for capturing non-financial value alongside return), while continuing to offer longer time horizons and a wider range of grant sizes.

Acknowledgments

Special thanks to David Lang, Reggie James, Christina Agapakis, Kristin Ellis, Sarah Drinkwater, and Yatu Espinosa for their thought partnership on this piece.

He has some thoughts about how often proposals succeed, and what will happen with the current move towards funding multi-year grants upfront rather than year-by-year.

Understanding and Modeling NIH Research Grant Success Rates and the Impact of “Multiyear” Funding

The National Institutes of Health funds the majority of the biomedical research in the United States. Perhaps the two most important parameters regarding NIH funding for researchers are the size of the NIH appropriation and the grant success rate (the probability of having a grant proposal funded). How are these two parameters related?

The NIH appropriation (in nominal dollars) over time is shown here.

This graph reveals the NIH budget “doubling” from 1998 to 2003, followed by a period of relatively flat funding, followed by steady, but more modest, increases starting in 2016.

For comparison, the success rates for Research Project Grants are shown here:

The success rate is defined as the number of competing grant applications funded divided by the number of applications reviewed in the same fiscal year.

The success rate did increase somewhat during the “doubling” but then it plummeted from > 0.30 (30%) to at or below 0.2 (20%) within a couple of years after the doubling ended. It has remained close to this level ever since, even with the increases in appropriation over the past decade.

Overall, these two parameters are negatively correlated with a Pearson correlation coefficient of -0.69.

Because NIH has increased grant sizes to correct for inflation (at least to some extent), one obvious adjustment is using the NIH appropriation in constant, rather than nominal, dollars. Corrections are made using the Biomedical Research and Development Price Index (BRDPI) rather than simple consumer inflation indices. The NIH appropriation in constant 1990 dollars is compared with that in nominal dollars below:

This graph reveals that the NIH appropriation in constant dollars actually peaked in 2003, dropped for about a decade, and then recovered almost back to the 2003 level.

The appropriation in constant dollars is still negatively correlated with the success rate data, but the correlation coefficient is somewhat less negative at -0.49.

The other key factor is the number of applications submitted and reviewed each year. These data are shown below:

This shows that the number of applications grew fairly steadily, and grew especially dramatically, not during the “doubling” but just after the doubling ended.

Not surprisingly, these two factors, the inflation-corrected NIH appropriation and the number of competing applications reviewed, account for the observed success rates with a correlation coefficient between the ratio of the inflation-corrected appropriation to the number of application and the observed success rates of +0.89.

A Model for Predicting Success Rates from Appropriations Data

While it is possible to make good estimates of the success rate given the appropriation (inflation-corrected) and the number of applications, the latter information is not available until NIH releases it well into the next fiscal year and, at that point, the success rate is also available.

It would be desirable to have a way of estimating the likely success rate given only the NIH appropriation, since that is the first parameter publicly available. Years ago, I developed such a model. Here, I describe that model and update it through fiscal year 2025.

The model has two components. The first deals with the number of applications expected in a given year. As noted above, the number of applications almost always increases over time but the increases tend to be larger a year or two after the rate of growth of the NIH appropriation decreases.

To capture this behavior, the number of applications was fit to a linear model based on the appropriations for the previous two years of the form:

Number_of_applications = c1*Appropriation_for_one_year_prior + c2*Appropriation_for_two_years_prior + c3

The expectation is that c1 would be negative and c2 would be positive to capture the fact that the rate of increase of the number of applications goes up after the growth in appropriations drops. Inflation-corrected appropriation values were used.

The number of applications predicted from this model is compared with the actual number of applications below:

The agreement is reasonable although the predicted and observed curves differ from one another more starting around fiscal year 2015. The model predicts large increases in applications starting in fiscal year 2022 which have not occurred.

The use of more complicated linear models does not result in significantly better agreement with the number of applications observed over the range from 1990 to 2024.

The second component is the number of competitive grants funded each year. These data are available from NIH, some months after the end of each fiscal year. The results are shown below:

This curve is rather similar to the NIH Appropriation in constant 1990 dollars shown in Figure 3. This is not surprising since appropriated funds are necessary for making grant awards and NIH has made attempts (albeit sometimes unevenly) to have grant sizes keep up with inflation. These curves are compared directly in Figure 7.

Thus, the inflation-corrected appropriation should allow a reasonable estimate of the number of competing grants funded.

However, there is one additional correction that better reflects reality and improves the accuracy of the model. NIH historically funds grants on an annual basis. For example, when NIH issues a four-year grant, they only pay for the first year out of the current year’s appropriation but commits to pay the additional years of the grant out of subsequent year’s appropriations. In essence, for each multi-year grant funded, NIH takes out a mortgage for the subsequent years of the grant. The funds are tied up until the project period of the grant ends and then the funds are again available for new and competing grants. This is summarized in Figure 8.

Thus, approximately 3/4th of the appropriated grants funds are already committed going into the beginning of a new fiscal year.

This phenomenon can be incorporated into a simple model that assumes that all grants are four years in duration and begins with an equal distribution of grants in each of the four years. The model then moves forward with new grants made based on the funds available from the new appropriation minus the amount of committed funds. The average grant size is assumed to increase with BRDPI inflation.

The results of this model are compared with those from the simpler, appropriation-based model in Figure 9.

The model does remarkably well, capturing the small but important ups and downs from year to year. The biggest variance between the model and the actual number of awards made occurs during the period from 2000 to 2006, that is, from the middle of the budget doubling to a few years after its completion. This variance is due to programmatic changes made by the NIH starting in the middle of the doubling.

With additional funds available, many NIH institutes started new programs, generally involving larger team science efforts with mechanisms such as U54 cooperative agreements. These awards were often several million dollars each. These new programs allowed NIH to experiment with new approaches to supporting science and prevented grant success rates to rise well above historical norms that may have been deemed to be politically problematic.

With models for the number of applications and the number of competing awards funded in hand, each based only on historical data about NIH appropriations and BRDPI inflation, we can now predict the research grant success rate. The results are shown in Figure 10.

The model does quite well, predicting the sharp drop in success rates following the NIH doubling and matching the smaller variations in subsequent years. The Peason correlation coefficient is 0.83. Overall, both the mean success rate (0.237 versus 0.236) and the standard deviation in success rates (0.050 versus 0.054) are predicted accurately.

This model is useful for predicting success rates once the appropriation for a new fiscal year is known. Moreover, if the observed result differs substantially from that from the model, then one should explore possible changes in policies and practices at NIH. Given the amount of turmoil at NIH over fiscal year 2025, it will be interesting to compare the actual results with those predicted from the model once NIH releases the official data. However, a recent change in policy that is anticipated in increase in importance in the current fiscal year must be considered.

Multiyear funding

As noted above, NIH generally funds grants on an incremental, annual basis. However, NIH has always “multiyear” or “fully funded” a small percentage of grants, taking all years of funding for the grant out of the appropriation for the year in which the grant is initially made. This practice was called out in the Congressional Justification for the fiscal year 2026 NIH budget, which included:

Traditionally, NIH research grants have been awarded for more than one year and funded incrementally; each year’s commitment is obligated from that year’s appropriation. Under this incremental funding approach, grants are classified as competing in the first year of award or renewal, and noncompeting in the remaining years of each award. Additionally, full funding has been provided up front for a limited number of grants and cooperative agreements as appropriate in special circumstances. Shifting to upfront funding for half of each year’s allocation for competing RPGs will increase NIH budget flexibility by no longer encumbering large portions of each year’s appropriation for the continuation of research projects that were initiated in previous years. As “legacy” noncompeting research projects phase out over the next few years, this shift in grants policy will make a greater portion of RPG funding available for new research projects each year.

As you can see from the discussion above, there is some truth to the argument about increasing budget flexibility. However, the transition from annual funding to multiyear funding creates problems. Suppose NIH funds a 4-year research grant using multiyear funding. This commits approximately 4 times as much funding as would have been committed had the grant been funded on traditional, annual basis. Thus, 3 other grants on the same size cannot be funded for each multiyear funded grant. This leads to a drop in the number of projects that can be funded with the corresponding drop in funded investigators.

The only real benefits to investigators and institutions from multiyear funding is that the funds can be spent faster if desired and there is no requirement for non-competitive renewal awards. However, these awards are historically awarded 98-99% of the time (and there are reasons when they are not) so this is not really much of a benefit and does give NIH less real oversight capabilities. The funds are not actually transferred from the treasury until they are spent, so there is a loss of cash flow to institutions as more spending authority sits in treasury accounts and not elsewhere.

It would be possible to responsibly manage the transition to multiyear funding by adding funds to the NIH appropriation during the transition years to pay for the potential loss of projects. But, it is noteworthy that the administration proposed this transition in the context of proposing a 40% cut in appropriations. This suggests that the rationale for multiyear funding is actually just a pretext for cutting funds to the extramural scientific enterprise and for other purposes, to be discussed below.

How much multiyear funding has been done in the past? Getting a clear definition of how NIH defines multi-year funding has not been straightforward. The simplest definition is that a grant in multiyear funded if the budget period is longer than 1 year. I use 12.1 months to avoid including grants that are a day or two longer than 1 year. An additional parameter of interest is the amount of funds committed beyond 1 year. This relates to the duration of multiyear funding grants. For example, the funds for 2-year multiyear funded grant could have been used to fund an annually funded grant of the same size whereas the funds for 4-year multiyear funded grant could have been used to fund three annually funded grants of the same size.

Let us first consider fiscal year 2024. The total amount of funded devoted to multiyear funded grants is $3.13B which represented 8.7% of the total NIH extramural commitments. Limiting consideration to new or competing grants, the total funds committed to multiyear funding grants was $1.40B which represented 16.2% of the total funds committed to new and competing grants. The percentage of funds committed beyond 1 year was 4.80% of the total extramural commitments.

We now turn to fiscal year 2025. The total amount of funded devoted to multiyear funded grants is $4.62B which represented 13.0% of the total extramural commitments. Limiting consideration to new or competing grants, the total funds committed to multiyear funding grants was $3.61B which represented 30.1% of the total funds committed to new and competing grants. The percentage of funds committed beyond 1 year was 7.75% of the total extramural commitments.

From these data, it is clear than NIH increased the amount of funds committed to multiyear funding for new and competitive grants by a substantial amount, more than a factor of 2 in terms of dollars and nearly a factor of 2 in terms of fraction of new and competing funding.

The differences are even more striking when examining grant mechanisms that were used for multiyear funding. In fiscal year 2024, the topic mechanisms (in terms of number of awards) were:

R21: 484

R15: 237

R03: 162

DP2: 108

RF1: 80

R01: 21

R35: 0

R21 and R03 grants are almost always 2 years in duration intended for projects limited in scope. R15 grants are 3-year grants intended to non-research-intensive institutions. DP2 grants are NIH Director New Innovator award which have generally been multiyear funded since they were created in the middle of a fiscal year when continued funding was not guaranteed. RF1 grants are explicitly a multiyear funding mechanism, implemented by administratively converting R01 grants. This mechanism has been largely used by the National Institute on Aging (NIA) in the context of rapidly growing appropriations. R01s are the “bread-and-butter” investigator-initiated research mechanism at NIH. R35 is a mechanism used primarily by the National Institute of General Medical Sciences (NIGMS) and the National Cancer Institute (NCI) for funding individual investigator’s research programs as opposed to projects.

For fiscal year 2025, the corresponding numbers are:

R21: 915

R15: 199

R03: 248

DP2: 102

RF1: 249

R01: 743

R35: 58

Most strikingly, the numbers of the bottom three “bread-and-butter” mechanisms increased from barely 100 to over 1000.

The Impact of Multiyear Funding

The previous model predicts that there would be 9,136 new and competitive renewal grants in fiscal year 2026. But, with the increased level of multiyear funding in fiscal year 2025, decreasing commitments, this number is increased to 9,751 with the assumption that no excess multiyear funding would occur in fiscal year 2026.

But, given that the administration has stated that it intends to use a substantial amount of multiyear funding, it is important to model the effects of this multiyear funding. For this purpose, we assume that multiyear funding includes an equal number of multiyear funded awards with durations of two years and four years.

With this assumption, we can estimate the number of new and competitive renewal grants as a function of the percentage of funds for competitive grants devoted to multiyear funding as shown in Figure 11.

The administration had originally stated that its intention was to use 50% of the funds available for new and competing grants. However, in the appropriations bill that passed the Senate appropriation subcommittee, there was an amendment introduced by Senator Capito from West Virginia and Senator Baldwin from Wisconsin that would have restricted the amount of multiyear funding in fiscal year 2026 to fiscal year 2024 levels. In the negotiations for the “minibus” appropriations bill that includes NIH but also some other agencies, the House and Senate conference modified this to restrict multiyear funding to fiscal year 2025 levels, apparently at the insistence of the Office of Management and Budget led by Russ Vought.

If we take the level of multiyear funding in fiscal year 2024 as the baseline included in the model, the percentage increase is 30.1 – 16.2 = 13.9%. From Figure 11, this corresponds to a loss of approximately 970 new and competing grants that would be funding in fiscal year 2026. This represents projects delayed or lost, investigators unfunded, and institutions stressed.

Note that Congress could mitigate this damage by increasing the NIH appropriation. My models yield estimates that a 4% increase in the NIH appropriation would enable multiyear funding to be used without a loss of the number of funded new and competing grants. However, even a flat budget results is an estimated 2.7% loss in purchasing power based on BRDPI. Thus, a 6.7% increase would enable both the loss of buying power due to inflation and a loss of the number of awards due to multiyear funding.

This push to inflict this damage on the scientific enterprise raises the question as to why this appears to be such a priority of the administration. Insight comes from Project 2025, of which Russ Vought was a chief architect. A section on NIH reads as follows:

Funding for scientific research should not be controlled by a small group of highly paid and unaccountable insiders at the NIH, many of whom stay in power for decades. The NIH monopoly on directing research should be broken. Term limits should be imposed on top career leaders at the NIH, and Congress should consider block granting NIH’s grants budget to states to fund their own scientific research. Nothing in this system would prevent several states from partnering to co-fund large research projects that require greater resources or impact larger regions. Likewise, the establishment of funding for scientific research at the state level does not preclude more modest federal funding through the National Institutes of Health: The two models are not mutually exclusive.

The second section speaks to pushing Congress to convert NIH, at least largely, into a block grant program. This transition would be facilitated by the NIH having a relatively clean slate with relatively few commitments going into some fiscal year. This would be facilitated by increased use of multiyear funding. A block grant would require the generation of systems for allocation of funds to the individual states, the development of a system for application submission and review at each state, ideally coordination of research across states, and so on. The NIH is certainly not perfect and thoughtful reforms would be welcome, but this ideologically-driven damage to what has been described by numerous legislators from both parties of the “crown jewel” of the federal government seems unwise.

Observations that this is the plan is provided by the first part of this section. At this point, through non-renewal of contracts, firings, reassignments, and coerced retirements, more than half of the 27 institute director positions are now open. The search processes to fill these positions differ dramatically from historical norms with no apparent input from expert scientists from outside the NIH and no substantial outreach to try to interest qualified candidates in applying for the positions.

Conclusion

The results from this analysis are summarized in Figure 12.

Success rates for NIH grant applications can be predicted relatively accurately from the history of NIH appropriations. Major drivers are the inflation-corrected appropriation and the number of competing applications which has tended to grow fairly monotonically, both in time of favorable and less favorable appropriations. A weakness in this analysis is the lack of data about the number of applications in fiscal year 2025 which is hard to predict given the unprecedented behavior by NIH in this year. Year-to-year commitments also play an important role. Funding more than one year of grants out of the appropriation for the year of the award reduces the number of new and competing awards that can be funded. Currently proposed policies are likely to lead to ~1000 (~10%) fewer new and competing grants being funded in fiscal year 2026 than would have been funded in years with historically normal levels of multiyear funding.

My friend Jordan Dworkin recently wrote an excellent piece titled, “How Much Should We Spend on Scientific Replication?” It is the first attempt to model the probability that funding replication studies will be more impactful than just funding new scientific studies.

I saw the piece ahead of time, of course, but after doing a little more rumination, I have a couple of thoughts that I should have offered earlier.

First, Dworkin says that we should prioritize replicating new studies, based on the typical pattern of how citations accumulate:

First is when the replication happens — the attention a paper receives, and the extent to which it drives follow-on research, are time-dependent. Papers tend to accrue ~2.5% of their total citations in the first year after publication, 7.5% in the second, and ~12% each in years 3-5; by the sixth year post-publication, the average paper has already received almost half of the direct attention it ever will. Because the capacity of a failed replication to reduce a study’s impact depends on when the replication occurs, we should not prioritize replicating studies that have already accrued a lot of citations — at that point, it may be too late to capture most of the replication’s value. Instead, we should aim to replicate papers that are likely to accrue many citations in the future.

I’m not sure I agree with this assumption that the value of replication is mainly in heading off citations early on in a paper’s lifetime.

The famous 2006 Alzheimer’s paper that turned out to be likely fraudulent was cited over 3,600 times according to Google Scholar. 414 of those citations occurred since 2021, starting 15 years after the paper was published. Here’s the historical pattern (and keep in mind that mid-2022 was when the potential fraud was uncovered):

I think it’s fair to say this was still an influential paper, 15+ years after publication, and the full extent of its influence wasn’t just in the direct citations but in all the follow-on papers and grants, many of which might not have cited the 2006 paper directly:

I think there’s something to the following stylized model of how a lot of science operates:

Paper A comes out in Year 1. It gets a fair bit of attention early on, because it does something new/surprising/useful. Papers B, C, and D then build upon it in the next year or 2. By 5 or 10 years out, Papers E through Z are now highly influential papers that build on the line of research started by Paper A. At this point, direct citations to Paper A start to drop off (unless someone just wants to be historically complete), because people are citing the dozens or hundreds of more recent papers that build upon it.

In this case, I think it could still be valuable to directly replicate Paper A with as much rigor as possible. If Paper A was wrong, it is possible that hundreds of papers in the last 10 years have been chasing after the wrong idea.

If the Alzheimer’s replication had happened earlier on, sure, that would have been even better. But the mere fact that time has elapsed and direct citations have dropped doesn’t necessarily mean that the paper is any less influential. It could be quite the contrary: The paper’s seminal status is now so embedded in the field that direct citations aren’t capturing anywhere near the amount of influence it has, and direct replication would still be valuable.

Here’s another wrinkle on citation patterns, from another famous paper: Langer’s 1975 psychology paper “The Illusion of Control.” (hat tip to Science Banana for this example).

Its citations over the years have been impressive, and even grew considerably in the 4th and 5th decades after publication!

But was it a reliable paper? Well, a replication paper in 2021 had over 10,800 participants (17 times the original sample size), and found that the original paper didn’t stand up.

According to Google Scholar, the replication paper has only been cited 42 times in 4 years (even while the original paper still gets cited 200 times per year). Which is disheartening.

Speaking of whether people cite a failed replication or the original study, Dworkin’s policy brief mentions the fact that the value of replications might depend on how often people pay attention to replications:

I agree with this passage to some extent, but where I differ is this: it assumes that the community’s response to a replication study is just a given fact to be assumed (i.e., a “35% reduction [in citations] after a few years”).

But how the community responds to replications isn’t just a fixed and static quantity that no one can possibly change going forward.

A large-scale replication initiative (at the NIH, for example) might not just fund replication studies and then let the citations settle out wherever they may. Instead, it might also include a focused effort to:

• Publicize the effects of replication studies via scientific societies and their newsletters

• Update PubMed to change any study’s webpage to prominently include the results of any replication

• Work with journals like Science and Nature to make sure that their own webpages are updated with replication results.

• Announce that the NIH will sharply discount any grant applicants who cite the original paper but not the replication, because they are obviously not up to speed on their own field.

An agency like NIH has many tools to get scientists to pay more attention to replications! The amount of attention paid to replications is not a fixed (and low) quantity that we have to take as given, thereby diminishing the impact of doing any replications at all.

Anyway, these are smaller quibbles in the grand scheme of things. Jordan Dworkin’s piece is excellent, and we need more of this in science/metascience.

From Jordan in response:

First off, thanks for engaging with the piece and thinking critically about it. You’ve raised some great points. We agree on many things, but there are a few I think are worth pushing back on.

Continued-influence and time-elapsed are both important

One key point that you raise is that the temporal aspect of my model should capture something more like “continued influence” rather than a strict “time elapsed” parameter. So if a paper is 15 years old but still getting hundreds of cites per year, there could still be lots of value in replicating it.

I agree with that, and it’s true that the default 15-year trajectory is a bit limiting. But the model doesn’t downweight the value of replicating old papers, per se; it downweights papers whose influence is mostly in the past. An older paper that is still accumulating citations at a high rate will show high expected future impact, and thus high replication ROI. Plugging in parameters that roughly correspond to the Lesné paper — moderate skepticism (say, 33% chance of failure) and 3600 citations after 15 years — still shows 9x returns for a replication at year 10.

But even granting that a Lesné replication at year 10 would have been valuable, it’s important not to discount timing’s role here. We knew that paper was impactful after 5 years, and if you assume a 5-year replication in the model instead of 10 then the ROI is 39x; if you plug in 3 years, it’s 59x.

So I agree that there are older papers that we should still replicate (and I think in many cases the model would still show those delayed replications as high-ROI). But in most of those cases we would have gotten substantially higher returns had we replicated them earlier, so finding those early opportunities should be a large focus of any new program.

Baked-in influence probably makes replications less impactful, not more

You acknowledge the timing point, but also argue, “the mere fact that time has elapsed and direct citations have dropped doesn’t necessarily mean that the paper is any less influential. It could be quite the contrary: The paper’s seminal status is now so embedded in the field that direct citations aren’t capturing anywhere near the amount of influence it has, and direct replication would still be immensely valuable.”

On the broad point about non-citation measures of influence, I agree that citations are limited as a tool here, and think that finding ways to source scientists’ perceptions of a paper’s influence (either instead of or in addition to metrics) would be a better approach for an actual program. But I think we might disagree on the implications of accumulated influence. I’d argue that the more baked-in a paper’s influence becomes, the less impactful a replication will be. Once there are hundreds of follow-on studies and a few subfields driven by a paper, un-ringing that bell takes far more than an independent replication.

In Lensé’s case, it took egregious and unambiguous fraud, layered on top of a slow accumulation of uncertainty based on years of small-scale failures to replicate. A single failed replication paper in 2022 would (in my opinion) have had a small fraction of the power of the fraud investigation, partly because the broader field was so developed that people might not have actually cared that much whether any individual oligomer held up.

An earlier replication, however, might have stood on more equal footing with the original findings. I think the Langer example supports this, in a way: by 2021 hundreds of papers per year were citing Langer as a foundational paper, and the replication couldn’t move the needle. Had the replication happened in 1996, when it was clear that the influence was rising rather than falling, but the scope of influence was still manageable, the trajectory might look very different today.

So if impact compounds in ways that are invisible to citation counts, and the influence-reducing ability of any given replication decreases as a function of a paper’s current influence, then my framework might even be conservative with respect to the importance of timing.

It is definitely true that there are non-citation measures of influence (hype, funding, trials) that might lead a simple model to discount the importance of replicating an older study. And I think that reducing the influence of incorrect foundational studies can be highly valuable if achieved. But I would caution that the presence of that hidden influence might suggest that replication is no longer sufficient for achieving those ends. The Lesné fact pattern — “egregious fraud, in the most widely-discussed scientific domain, with a paper published in Nature and an exposé published in Science” — is not one that most NIH-funded replications will be able to work with.

Maybe we should replicate the descendents

For similar reasons to those above, I’m a bit torn on your stylized model. If follow-on Papers E through Z are cannibalizing citations from Paper A, I think there are cases when it would genuinely be higher impact to replicate those studies rather than Paper A; if Paper A fails to replicate, but I still trust Papers E and F, will I actually change my research?

I admit that the Lesné example cuts against this a bit. Perhaps there’s a distinction between papers whose influence operates mostly at the level of “creating hype and funding for a flawed subfield” and those whose influence operates by “producing a specific finding that lots of others try to build on.” In the former case, replicating the focal paper is probably higher value even if the follow-on papers are cannibalizing citations. In the latter, if papers E through Z are becoming more widely cited in their own right, I think replicating them could genuinely be higher impact. My guess is the latter situation is more common, but instances of the former might be particularly high leverage.

Increasing attention is critical

On your last point about not treating community response as static, we’re in agreement. The model treats the reduction in citations as a fixed parameter mostly because that’s the ecosystem that any new replication program will exist within, and we should be clear-eyed about the impact those replications will or will not have.

But you’re right that 35% is not a ceiling, and it would be very valuable for NIH to try some levers for getting scientists to pay more attention to replications. I would be excited to see, for example, an effort to more cleanly link (high quality) replications to original papers, and I think your PubMed idea is a clever and tractable one.

FROM STUART IN REPLY:

Thanks for these many thoughtful remarks, Jordan! I don’t think we’re that far apart, if at all.

I agree that even if older papers still deserve replication from time to time, it would have been more valuable to replicate them earlier rather than later.

I agree that if a paper’s findings have become embedded in a literature of hundreds or thousands of follow-on papers, then even if the entire literature is spurious, merely replicating the original paper might not be anywhere enough to overturn that literature.

I agree that replicating follow-on papers might well be more high-impact.

And we both agree that NIH and others can do much more to raise the prominence and salience of replication studies.

So, here’s to both of us raising a glass towards more replication at NIH!

A PDF version of this piece is available here.

Introduction: An Opportunity in Plain Sight

While the US pharmaceutical industry excels at developing novel, patentable compounds, world-class trials of low-exclusivity treatments, such as dietary supplements and off-patent medications, are pursued neither by industry nor the NIH. In addition to missing opportunities to improve health for all Americans, this trial gap costs the public and the federal government hundreds of billions of dollars a year.

Because supplements and off-patent medications do not have economic mechanisms that lead pharmaceutical companies to run Phase III trials, there is an urgent need to develop reliable, large-scale evidence so that patients and providers can access these treatments.

Americans currently spend ~$70 billion annually on supplements, often acting on fragmented or inconclusive data. Simultaneously, hundreds of off-patent medications with known safety profiles sit on the shelf, untested for new indications because manufacturers do not have the legal exclusivity needed to justify the cost of large, conclusive Phase III trials, even in areas of very high unmet need.

In Senate testimony in 2020, Jay Bhattacharya addressed this ‘market failure’ directly:

“For drugs and therapies on patent, a patent holder has a strong interest in running randomized evaluations and navigating the drug through the FDA’s approval process. By contrast, for drugs and therapies with no patent holder, no one has much interest in funding expensive randomized trials or working assiduously to move through the FDA regulatory process for rapid approval (or even slow approval).”

To solve this gap and open up new treatments and dramatic cost savings, we propose the creation of an NIH High-Leverage Trials (HILT) Program for off-patent drug repurposing and supplements. HILT would fund and run definitive large scale trials and, when appropriate, advance regulatory approvals to ensure broad patient access.

Cutting Waste

There are already scattered examples that demonstrate the cost-saving potential of NIH running trials that industry players will not study.

In 2008, the National Eye Institute (NEI) funded the Comparison of AMD Treatments Trials (CATT) to compare Lucentis (macular degeneration treatment costing ~$2,000 per dose) vs Avastin (a structurally similar treatment, also under patent, but costing ~$50 per dose for this use). This is the type of head-to-head comparison that no commercial player had an incentive to run. The trial showed both drugs were equally effective, which transformed prescribing practices toward bevacizumab and OCT-guided therapy, and resulted in cumulative savings for Medicare Part B estimated at $40 billion (for context, NIH’s entire annual budget is ~$50B).

By selecting trials with similar potential for high economic impact, HILT could generate savings for the general public, private payers, and public payers that would pay for itself many times over.

Improving Safety

Supplements are lightly regulated, which has both benefits and risks for patients. Access is generally good but high quality research is thin. A 2015 study showed that adverse events related to dietary supplements cause approximately 23,000 emergency department visits annually. Large-scale safety and efficacy trials on supplements will help guide the public towards safer and more effective supplements and clarify effective dose levels.

HILT would not promote supplements or repurposed drugs, but rather generate transparent and high-quality evidence, whether that evidence is positive, negative, or safety related. In high‑risk contexts (such as probiotics in preterm infants), regulators have raised concerns about product quality and potential infectious complications, even though there are also potential indications of benefits. Well designed, large-scale trials can resolve safety and efficacy questions while offering clarity on dosing and quality.

Better Treatments

When a prescription medication approaches the end of its patent and exclusivity life, research investment vanishes, even if the drug has lots of promise for additional uses. Budish, Durvasula, et al. recently quantified this repurposing gap, demonstrating that the loss of exclusivity for a medicine leads to a near-total cessation of clinical trials for new indications, due to the collapse of private incentives (other research concurs).

The market fails because these trials are very expensive, and even if the trial is successful and the company gets a 3-year indication-specific labelling exclusivity from the FDA (505(b)(2)), they cannot prevent other generic versions of the drug from being prescribed off-label for this new indication, which kills their potential profits. Therefore no private companies invest in these trials.

Their modeling of the repurposing cliff estimates that loss of exclusivity has resulted in 200-800 missing new uses for existing drugs.

The authors calculate that the social cost of these lost opportunities is on the order of several trillion dollars. This market failure leaves patients with higher-risk or lower-efficacy products while potentially superior low-cost treatments are never tested and therefore never available or covered for patients. Solving this gap would bring new treatments to patients with dramatically lower costs and increased safety.

Creating a Solution

HILT will function as a funding, research, and regulatory advancement body, modeled in part on the successful Best Pharmaceuticals for Children Act (BPCA), in which NIH (via NICHD/PTN) identifies off‑patent drugs, funds or sponsors trials, and submits findings to FDA for label changes. HILT would be an adult‑population analog for supplements and off‑patent therapeutics. HILT would build on and coordinate with existing NIH efforts (e.g., ODS and NCATS) while adding standing Phase III operations and end-to-end translation capacity (FDA + payer alignment).

HILT will:

● Generate “gold standard” evidence by moving promising unpatented compounds through large, well-designed Phase III trials.

● Define and study dosing and reference standards for supplements in trials, to provide clear consumer guidance and ensure that trial results are actionable.

● Validate low-cost alternatives to expensive therapeutics, providing the data necessary for CMS and private payers to reimburse cost-effective treatments.

● Serve as the non-commercial IND holder as needed, navigating regulatory pathways to enable access and coverage for patients.

Crucially, HILT would not increase or alter regulatory burdens on the private sector. Instead, it fills a void that the industry is not able to address, providing cost savings and better treatments to all Americans.

Mission and Operations

To accomplish its mission, HILT would:

● Develop staff and organizational expertise in the identification of supplements and off-patent medicines with a high expected-ROI from advancing large scale trials.

● Take advantage of extensive real-world safety data for many supplements and repurposed medications to design and fund or co-fund clinical trials with dramatically lower cost structures (e.g., telehealth-based designs).

● Create internal research programs and develop expertise in the unique economic and IP dynamics of supplements and off-patent therapeutics.

● Provide grant funding to non-commercial efforts to run trials with the ability to operate in the role of a program ‘sponsor’ in advancing trials and navigating existing regulatory pathways at the FDA.

● Establish partnership programs with industry to provide strategic or complementary funding that enables privately run large-scale trials.

● Collaborate with other NIH Institutes and programs to draw on disease-specific expertise when evaluating opportunities.

● Identify opportunities to run large-scale head-to-head trials (e.g., comparing multiple generic SSRIs or different forms of Vitamin D) that industry is often disincentivized from running. Precedents here include BPCA, Pediatric Trials Network, and PCORI.

● Launch research and develop expertise in quality testing and contaminants. Fund and publish methods, partner with established third‑party certifiers (USP, NSF) and standards bodies, and create data standards.

● Align evidence generation with payer coverage to ensure patient access and availability.

Creation of HILT would likely build on and coordinate with existing programs at NIH’s Office of Dietary Supplements (ODS) and NCATS’s repurposing programs. These existing efforts provide benefits to patients but are not positioned to advance research through large scale trials and ensure that scientific, patient, consumer, manufacturer, prescriber, and payer incentives are aligned end-to-end.

Program Scale and Economics

HILT’s goal is to minimize cost per decisive answer by focusing on compounds with extensive human safety history and by using pragmatic, embedded, and decentralized trial designs when possible. Compared with traditional pivotal drug development, which requires expensive infrastructure to evaluate the safety of novel compounds, this enables definitive trials to be run at substantially lower costs while preserving rigor.

HILT will be evaluated on output and outcomes: trials launched, cost per answer, downstream changes in coverage/labeling, new indications approved, impacts on consumer supplement spending patterns, and measured payer savings where utilization shifts to lower‑cost generic options.

Establishing Legislation and Positioning for HILT

To be successful, the capabilities of HILT program should include:

● A permanent program office with staff aligned towards ROI and cost-per-decisive-answer (rather than a more traditional disease-area publication focus).

● Capacity to run Phase III pivotal trials with an orientation towards innovation on cost and approach, taking advantage of the advantageous safety data available for most of these candidates. This will require standing trial-operations capacity along with flexibility for external grantmaking, as appropriate.

● Regulatory capacity to act as a drug sponsor, including FDA engagement and submission. For successful therapeutics, the goal should be to achieve patient access and coverage, not simply publish evidence. The program should coordinate with CMS and private payers so evidence generation is aligned to eventual coverage decisions.

There are several ways that this program could be created and positioned for success within NIH.

These include:

● Expand BPCA: Expand BPCA authorities beyond pediatrics to a) cover repurposing research at all ages and b) add a focus on supplements. Potentially a straightforward political path that builds on the success of BPCA.

● New Institute: Create a new NIH institute, perhaps a National Institute of Supplements and Repurposing (NISAR). This would be a strong home for the program, enabling institutional capacity and expertise to develop and would potentially absorb some existing NIH programs. However, it may be a bigger lift politically.

● New NIH Center: Establish a joint Center for Supplements and Repurposed Therapeutics within NIH co‑led by ODS and NCATS, with a dedicated budget line and explicit authority to a) act as non‑commercial sponsor / IND holder, and b) submit data to FDA to support label changes or qualified claims.

Other pathways to establishing and situating this program are also possible.

Opportunities for dramatically cheaper clinical trials

Supplements and off-patent medications often have decades of public use and well understood safety profiles. This is a massive advantage relative to new drugs following the typical biotech and pharma approval pathway. Because extensive real-world safety history and/or previous pivotal trials reduce safety uncertainty, clinical trials for new indications have multiple opportunities to succeed at lower costs.

In collaboration with the FDA, HILT can develop experience and expertise in novel trial design models that are rare in traditional drug development. This will increase the speed and cost efficiency of HILT’s programs and, as importantly, will create models for running Phase III trials more efficiently. Pharmaceutical development is a highly risk-averse industry and avoids innovation in pivotal trials to reduce regulatory complexity and risks of failure. Pharma is much more likely to pursue lower cost trial innovations at Phase III if precedents are demonstrated. HILT would be perfectly positioned to play this role.

Strategies for cost efficient clinical trials may include:

1. Embedded EHR-based randomized trials (RRCTs). This approach runs a trial “inside” routine health care. Patients are randomized within large health systems or networks, and most outcomes are pulled automatically from existing electronic health records and insurance claims (like hospitalizations or medication changes), rather than collecting lots of additional study-specific data for each patient at each site. This approach can help test repurposed drugs or supplements in real-world care. NIH’s Collaboratory already has practical guides that show how to design and run these trials.

2. Decentralized telehealth trials. Patients interact remotely with trial staff and receive study medicines by mail or at local pharmacies. This avoids the massive per-patient costs charged by clinical research facilities– costs that often make up the majority of a clinical trial program budget. Several large decentralized trials demonstrated feasibility of this model during the pandemic lockdowns, but pharma has been hesitant to pursue this model due to high risk-aversion. Approaches like video consent, home delivery of standardized product, ePROs, wearables, use of local commercial blood labs, and mail-in tests when applicable can all enable cost savings. NIH Collaboratory guidance supports these methods.

3. Multi‑arm, multi‑stage (MAMS) or platform trials with shared controls. Compare several generic medications within an indication in one platform, drop futility arms early (for example, head-to-head SSRI or supplement trials). A single master protocol lets you reuse the same operational setup and monitoring for many comparisons. A Practical Review of Adaptive Platform Trials.

4. Factorial designs to disentangle combos. 2×2 or 2×2×2 designs can test components and interactions without dramatically scaling participant numbers, for example, in migraine or metabolic supplement combinations.

5. Cluster/stepped‑wedge trials. Trials where the unit is a setting rather than a participant. For example, to study artificial food dyes in schools, randomize schools or districts, rotate implementation (stepped‑wedge), and use validated classroom behavior metrics. Can be much cheaper than typical site‑based RCTs.

6. Coverage‑with‑Evidence Development (CED). For candidates with plausible payer coverage, co‑design outcomes with CMS/private payers so positive trials can flow into CED or coverage updates.

7. Create a BPCA-style priority list. Formalize a “HILT priority list” for adult therapeutics (mirroring BPCA §409I), publish targets annually, and collaborate with FDA on trial and evidence pathways. EveryCure is a non-profit leader in therapeutic repurposing and has a multi-year contract with ARPA-H. They may be in a position to advise on trial opportunities and priority decision making.

HILT is distinct from existing agencies and programs

HILT’s role of bringing supplements and off-patent medications through Phase III trials and potentially regulatory approval is not accomplished by existing programs.

● BPCA (Best Pharmaceuticals for Children Act, NIH/NICHD & Pediatric Trials Network): an important pediatric-only precedent in which NIH identifies priority off‑patent drugs, sponsors or funds studies, and supports FDA label updates. BPCA is limited to children and repurposed therapeutics, whereas the goal of HILT is to expand these mechanisms to adults, include supplements, and pursue larger trials for novel indications.

● ODS (NIH): coordinates supplement research, funds methods/standards (AMRM), databases (DSLD/DSID), and co‑funds grants, but it’s not set up to run or sponsor phase‑3‑scale trials nor to pursue label changes. Without these functions, it has limited impact for patients. (Office of Dietary Supplements)

● FDA exclusivity programs: existing FDA incentive mechanisms which apply to non-patented drugs, such as the 3-year exclusivity for new clinical investigations, and 7-year orphan exclusivity, are insufficient and, as empirical studies clearly show, have not been able to address the repurposing gap. Even with marketing exclusivity, off-label use and substitution undermines the value capture for private sponsors. Incentive-only approaches also underproduce studies with high public ROI, like head‑to‑head comparisons.

● NCCIH & the NIH Pragmatic Trials Collaboratory: this work focuses on pragmatic trial methods and embedded trials across health systems, not a vertical mission to take supplements/off‑patents through labeling or payer alignment. (NIH Pragmatic Trials Collaboratory)

● NCATS: has repurposing programs (e.g., New Therapeutic Uses) and convened an FDA workshop on off‑patent repurposing that highlighted the lack of ROI for off‑patents, but NCATS does not run a standing program to sponsor and complete large scale trials for generics or supplements.

● PCORI: comparative‑effectiveness mission is complementary, but its scope is not designed to sponsor pivotal efficacy trials intended to establish new indications. (pcori.org)

● AHRQ: generates evidence via systematic reviews and methods for comparative effectiveness, not by sponsoring interventional trials to labeling. See Effective Health Care (EHC) Program.

● VA Cooperative Studies Program (CSP) runs large multicenter trials, but its remit is Veteran‑focused and not organized around supplements/off‑patent therapeutics or market‑access alignment for the general US population. (VA CSP)

● CMS can tie coverage to studies via Coverage with Evidence Development (CED), but CMS does not fund or sponsor those trials.

Questions and Answers

Is HILT a new regulator?

No, HILT would not regulate products. It funds and sponsors trials, develops methods and standards, and can hold an IND when needed to enable studies. Regulatory decisions remain with FDA and coverage decisions remain with payers/CMS.

Are there precedents for this approach?

Best Pharmaceuticals for Children Act (BPCA) at NIH (NICHD/PTN) identifies off‑patent priorities, funds/sponsors trials, and submits to FDA for label changes. HILT would be similar but for the adult‑population, covering both supplements and off‑patents. HILT’s scope and budget would need to be substantially larger than the NIH BPCA program (which is ~$25M/year), because BPCA’s off-patent pathway is generally oriented toward pediatric labeling gaps for drugs already approved for adults in the same or similar indications. This means BPCA trials have relatively small patient numbers and focus on dosing/PK and more narrow safety and effectiveness questions, rather than large, adult-style pivotal efficacy trials.

How will HILT choose what to study?

HILT would develop a BPCA‑style priority list, based on factors like:

● Public‑health ROI, including disease burden, affordability, and access.

● Potential cost savings to the government and public.

● Safety profile, such as years of real‑world use or existing pivotal trial data.

● Biological plausibility and early‑phase or observational signals.

● Readiness for large, pragmatic RCTs and coverage alignment.

● Collaboration potential with disease‑specific NIH Institutes and payers.

Will HILT “pick winners” or crowd out private investment?

HILT would fill the market gaps where private ROI is too low to fund definitive trials. It would co‑fund and partner with industry when practical (e.g., shared controls, platform trials). It would not take on trials that industry is incentivized to pursue. And for the public, but not for industry, negative or null trials are very valuable: they reduce waste and protect patients.

How will evidence move into labels and coverage?

Like a drug sponsor, HILT will engage with the FDA on trial design and operations, including guidance supporting decentralized trial elements and other cost-savings approaches where appropriate. It will co‑design with payers/CMS for Coverage with Evidence Development (CED) or routine coverage if results are definitive. For labeling on generics, legislation creating HILT could direct FDA to establish a pathway for “harmonized label updates” based on HILT-sponsored evidence, allowing the agency to update the reference label and have generic manufacturers adopt those changes without incurring new product-liability risk. Alternatively, HILT could serve as the holder of a public-health label that generic manufacturers may reference, similar to how they currently reference the original approved drug.

How will HILT build public trust?

It will be essential for HILT to select projects that have clear ROI for the public and run transparent, gold-standard trials. Because HILT will be a public agency rather than a private drug company, there is an opportunity for far more transparency of process and data than is typical in drug development. Public protocols and SAPs, trial registration, data‑sharing plans, and rapid results reporting, including negative trials will all be publicly available. Controls like COI firewalls for advisory panels and multisector input with scientific independence in decisions will be put in place.

What will HILT not do?

HILT will not create new regulations, set marketing policy, or police the retail market. Nor will it serve as a channel for proprietary promotion of drugs or supplements.

How would HILT support understudied issues related to food additives, nutrition, and toxin exposure?

HILT will be in a position to run rigorous trials on issues such as artificial food dyes and behavior, prenatal nutrition, or environmental‑exposure‑adjacent questions where product quality and safety are central. Publishing transparent methods and results will be essential to counter confusion and reduce misinformation, as will staying nonpartisan and science‑first, focused on outcomes not industry.

How will success be measured?

Because HILT has an ROI orientation, both for health and cost impacts, it will be relatively easy to review the success of the program over time. Indicators will include:

● Regulatory outcomes, such as FDA label or qualified claim change.

● Economic outcomes, such as cost per answer, budgetary or public savings, and improved adherence to low‑cost therapies.

● Coverage outcomes, such as CMS/private coverage decisions.

● Clinical/health outcomes (fewer hospitalizations, improved function, maternal‑infant metrics).

How can HILT generate economic value?

Compared with NME development, HILT’s focus on known‑safety drug and supplement candidates plus pragmatic trial designs can deliver decisive answers at lower cost per decision. Even a few high‑impact wins could generate substantial public ROI by improving access to low‑cost, safe options and curbing public spending on ineffective ones.

EXAMPLE TRIALS HILT COULD PURSUE

● Low-dose lithium orotate for Alzheimer’s disease and prevention (elevated biomarkers / MCI / early AD) Lithium orotate shows encouraging signals mechanistically, from recent animal studies, and from small human trials. HILT could define a dose range and run large pragmatic RCTs in high risk patients / MCI / early AD with safety labs and cognition/biomarker endpoints. Relevant evidence: Nature 2025 study & NIH summary, MCI RCT overview (Forlenza et al.).

● Melatonin for Insomnia (long‑term safety, efficacy, dosing/timing) Melatonin is widely used but doses vary by orders of magnitude. A recent American Heart Association 2025 abstract raised a possible, and contested, heart-failure signal from prolonged use. HILT could run a 12–24 month RCT comparing dose levels, immediate vs extended release, and safety. Relevant evidence: AASM adult insomnia guideline (2017), PR melatonin 2 mg RCTs in adults ≥55, JAMA melatonin-gummy mislabeling study, AHA 2025 retrospective.

● Low-dose naltrexone (LDN) for Long COVID (fatigue, pain, neurocognitive symptoms) LDN is widely used off-label with a well-understood safety profile, but no completed, definitive RCTs exist yet in Long COVID. HILT could study a standard formulation, phenotype responders, and run multi-site RCTs with patient-relevant outcomes. Relevant evidence: Systematic review (no completed RCTs yet), Ongoing randomized trial protocol.

● Ivabradine for POTS (including post-viral POTS) Ivabradine, which goes off-patent in 2026-27, improved heart rate and quality of life in a randomized crossover trial for hyperadrenergic POTS, but broader, pragmatic data are needed. HILT could fund confirmatory trials (including non-hyperadrenergic phenotypes) and generate coverage guidance. Relevant evidence: JACC randomized crossover trial summary, Full trial article.

● Metformin (early treatment) to prevent Long COVID An outpatient RCT found that starting metformin early in acute COVID reduced subsequent Long COVID incidence, replication and implementation evidence are needed. HILT could sponsor broader, real-world trials and translate results into labeling/coverage updates. Relevant evidence: Lancet Infectious Diseases RCT (COVID-OUT), JAMA Intern Med follow-up on acute recovery.

● Sulforaphane for Autism (core/behavioral symptoms) Several RCTs signal improvements in behavior and function, but products vary widely in active content. HILT could run a large Phase III with long-term follow-up. Relevant evidence: PNAS RCT (2014), Molecular Autism RCT (2021).

● N-acetylcysteine (NAC) for Autism (irritability/behavior) Multiple small RCTs suggest benefit for irritability with good tolerability, but dosing and product quality are inconsistent. HILT could select a dose/formulation and run a confirmatory Phase III. Relevant evidence: Hardan et al. RCT (Biol Psychiatry), RCT in youth with ASD (Molecular Autism).

● Artificial food dye impact on child behavior (hyperactivity/inattention) Meta-analyses support effects in susceptible children, yet policy and practice vary. HILT could run cluster-randomized school trials of dye removal with validated behavior measures and standardized natural colorants. Relevant evidence: Child & Adolescent Psychiatry, 2012 meta-analysis, Schab & Trinh 2004 meta-analysis.

● Berberine for dysglycemia/metabolic health (including PCOS and T2D) Berberine shows glucose and lipid improvements across RCTs but comparative data vs metformin are limited. HILT could study standardized actives (and contaminants) and run large head-to-head trials in metformin-intolerant or high-risk groups. Relevant evidence: Meta-analysis of RCTs in T2D, Randomized trial of berberine-ursodeoxycholate in T2D.

● Choline in pregnancy (maternal–infant cognition) Randomized feeding studies show higher-dose choline in late pregnancy improves infant information-processing speed, but optimal dosing has not been studied at scale. HILT could fund large, stratified trials that would support product standards and coverage pathways. Relevant evidence: FASEB Journal RCT (Caudill et al.), Follow-up on sustained attention.

● Iodine in pregnancy (neurodevelopment in mild–moderate deficiency) Iodine sufficiency is essential for development but effects of supplementation in mildly deficient populations are not sufficiently understood. HILT could run large trials with cognitive outcomes. Relevant evidence: WHO commentary & evidence review, ATA pregnancy thyroid guideline synopsis.

● Low-dose naltrexone for centralized chronic pain (e.g., fibromyalgia) Results from existing studies have been mixed. HILT could do dose-ranging and responder-enriched trials. Relevant evidence: Lancet Rheumatology FINAL trial (6 mg), Younger et al. randomized crossover pilot.

● Pharmaceutical-grade chondroitin sulfate for knee osteoarthritis Efficacy appears to depend on prescription-grade CS, robust RCTs show non-inferiority vs celecoxib, while lower-grade products yield inconsistent results. HILT could propose quality specs and run coverage-oriented trials. Relevant evidence: CONCEPT RCT (CS vs celecoxib vs placebo), MOVES non-inferiority trial (CS+glucosamine vs celecoxib).

● Omega-3 (EPA-focused) adjunct for depression Americans spend ~$2.5B per year on fish oil supplements. Evidence and international practice guidance support high EPA omega-3s as an adjunct in MDD, but dose/formulation quality vary. HILT could study EPA content effects and fund pragmatic adjunctive trials with biomarker stratification. Relevant evidence: ISNPR practice guideline (2019), EPA-focused meta-analysis.

● Saffron (Crocus sativus) adjunct for mild–moderate depression Multiple RCTs/meta-analyses suggest saffron may be comparable to SSRIs for mild–moderate depression with favorable tolerability. US-grade trials are sparse. Relevant evidence: Nutrition Reviews 2024 meta-analysis vs SSRIs, Umbrella/meta-analysis.

● Myo-inositol for Polycystic Ovary Syndrome (PCOS), ovulation/insulin resistance Myo-inositol is widely used with signals of benefit for metabolic and reproductive outcomes, but products/doses vary and head-to-head data vs metformin or MI+DCI ratios are inconsistent, HILT could select dosing (e.g., MI vs MI+DCI ratios), and run large head-to-head trials vs metformin with live-birth endpoints. Relevant evidence: Journal of Clinical Endocrinology & Metabolism (2024), Reproductive Biology and Endocrinology (2023)

● Psyllium (soluble fiber) for LDL lowering and cardiometabolic risk Psyllium lowers LDL modestly, but product quality and dosing vary, HILT could test dose–response RCTs against ezetimibe/low-dose statin backbones. Relevant evidence: eCFR (2025), American Journal of Preventive Cardiology (2020)

● Coenzyme Q10 for statin-associated muscle symptoms (SAMS) CoQ10 is widely tried for SAMS with mixed RCT meta-analyses, HILT could harmonize endpoint definitions (pain, CK, adherence) and run large, pre-specified trials in statin-intolerant cohorts. Relevant evidence: Atherosclerosis (2020), DOAJ (2025)

● Magnesium/riboflavin/CoQ10 for migraine prevention These nutraceuticals have “possibly/probably effective” signals but heterogeneous formulations and small trials, HILT could compare forms (e.g., magnesium citrate vs oxide), doses, and run head-to-head vs topiramate/candesartan with uniform outcomes. Relevant evidence: Neurology (2012), American Migraine Foundation (2024)

● Taurine for metabolic syndrome/cardiometabolic risk Meta-analyses suggest improvements in BP, glucose, and lipids but long-term outcomes unknown. HILT could study standardized taurine and test cardiometabolic endpoints and safety. Relevant evidence: Nature Portfolio (2024), Nutrients (2025)

● Glycine for sleep quality Small crossover RCTs show improved sleep quality with bedtime glycine. HILT could validate dose–response, safety, and compare vs melatonin and CBT-I. Relevant evidence: Sleep and Biological Rhythms (2007), Pharmacology Biochemistry and Behavior (2012)

● Vitamin D for fracture prevention in community-dwelling adults Large RCTs show no fracture benefit of routine vitamin D supplementation in generally replete adults, HILT could fund targeted trials by deficiency status/age/falls risk and harmonized dosing. Relevant evidence: New England Journal of Medicine (2022)

● Saw palmetto for BPH/LUTS Escalating-dose RCT showed no benefit vs placebo. HILT could definitively test standardized extracts or help retire ineffective products via negative Phase IIIs. Relevant evidence: New England Journal of Medicine (2012)

● Ginkgo biloba for dementia prevention GEMS found no prevention benefit. HILT could run conclusive negative trials (or targeted biomarker subsets) using standardized extracts. Relevant evidence: JAMA (2008)

● D-mannose for recurrent UTI prevention A recent multicenter RCT found no benefit vs placebo. HILT could settle dosing/subgroup questions (e.g., post-coital prophylaxis) or help deprecate use. Relevant evidence: JAMA Internal Medicine (2024)

● Ashwagandha for stress/anxiety Meta-analyses suggest reduced perceived stress/anxiety, but hepatotoxicity case reports exist. HILT could study standardized withanolide content, investigate dose/leaf-vs-root extracts, and run safety-focused RCTs. Relevant evidence: BJPsych Open (2021), Pharmaceuticals (2023)

Earlier this week, the House and Senate appropriations committees released a bipartisan bill on NIH funding, along with a Joint Explanatory Statement that has a few paragraphs of metascience interest. Note as well that the House and Senate both had already released language re: NIH that is still in place unless specifically overridden by the Joint Explanatory Statement.

Let’s dig in:

Indirect Costs

The Joint Explanatory Statement has some stern language dictating that neither NIH nor any other federal agency can change how indirect costs are calculated, or are they even allowed to issue a notice of proposed rulemaking that would attempt to do so. This is the most stringent language on indirect costs that I’ve ever seen:

Indirect Cost Rates.-The agreement recognizes that indirect cost recovery has been essential for supporting research at universities, nonprofit laboratories, medical centers and other entities eligible for Federal research awards and is key to sustaining U.S. leadership in scientific research and technological innovation. The agreement acknowledges that there is room for improvement in the system used to identify and recover indirect cost rates under the Uniform Grant Guidance, particularly with respect to the need for greater transparency into these costs. Various models have been suggested to achieve these improvements, including the Financial Accountability in Research (FAIR) model advanced by the Joint Associations Group on Indirect Costs (JAG), which the Committees believe merit further consideration. Therefore, the agreement directs the departments and agencies funded in the Act to engage in discussions with the Committees on proposals to achieve these improvements, including onthe FAIR model. Under this agreement, neither NIH, nor any other department or agency, may develop or implement any policy, guidance, or rule, including publication of a notice of proposed rulemaking, that would alter the manner in which negotiated indirect cost rates have been implemented and applied under NIH regulations, as those regulations were in effect during the third quarter of fiscal year 2017.

I assume that NSF, DOE, DOD, etc. are listening . . .

Fund the Person, Not the Project

The House report had this language, which remains in place:

Experimental Research.--The Committee recognizes the success of the NIGMS Maximizing Investigators’ Research Award program [R35 grants] and encourages the NIH to continue expanding similar experimental research opportunities to other institutes.

As well, the Senate report had this language about the same program:

Fund the Person, Not the Project.—While many labs are funded by R01-equivalent grants, the R35 mechanism arguably allows scientists more flexibility and freedom to pursue the most meritorious science. The Committee looks forward to reviewing NIH’s plans for expanding the R35 along with its plans for evaluating the impact on scientific progress, as directed in the Consolidated Appropriations Act, 2024 (Public Law 118–122).

Hopefully NIH can make some progress in this area. The R35/MIRA program is a great way to fund creative scientists with more flexibility to follow new ideas. The NIH should be using this program more widely.

Replication

As discussed here earlier, the House report had some ambitious language plus a budget of $100 million for replication at NIH. The new Joint Explanatory Statement rejects that House language in favor of the Senate language:

Replication Experiments and/or Fraud Detection.-In lieu of the amount and directive under this heading in House Report 119-271, the agreement supports NIH efforts to fund replication experiments on significant lines of research and continues the briefing directive under the heading "Replication and Reproducibility Experiments" under Senate Report 119-55.

The Senate language, unfortunately, is less ambitious:

Replication and Reproducibility Experiments.—The Reproducibility Project in Cancer Biology showed the difficulty in replicating cancer biology studies published in top journals. Given the importance of reproducibility in scientific research, the Committee encourages NIH to establish a program to fund replication experiments on significant lines of research. The Committee directs NIH to brief the Committee within 180 days of enactment on the reproducibility efforts NIH will undertake in fiscal years 2026 and 2027.

No more $100m. Also, the Committee merely “encourages” NIH to act, which is language that agencies routinely treat as optional, although I doubt that will happen in this case (Jay Bhattacharya is very much in favor of replication).

Restructuring NIH

There have been various proposals to reorganized and streamline NIH’s many components, but the Senate report basically rejects those ideas unless the proper statutory process is followed:

Restructuring NIH.—The Committee notes that Congress established 24 NIH ICs in statute through section 401 of the Public Health Service Act (Public Law 106–525). The Committee further notes that section 401 of the Public Health Service Act (Public Law 106–525) requires that the Secretary provide the HELP Committee and the Committee on Energy and Commerce of the House of Representatives 180 days written notice of any determination to restructure or reorganize the functions of NIH’s ICs, which the Committees have not received. The Committee commends NIH for reconvening the Scientific Management Review Board [SMRB], as directed in the fiscal year 2024 appropriations Act, to review the overall research portfolio of the agency and advise on the use of organizational authorities, including eliminating ICs, creating new ones, and reorganizing existing structures. NIH is directed to provide a report to the Committee no later than 30 days after enactment on SMRB activities, and an annual report on SMRB plans and activities thereafter.

The Burden of Bureaucracy

The Senate report reiterates earlier language on this point:

Reducing the Administrative Burden on Researchers.—The Committee remains concerned about the status of NIH’s implementation plans following a 2019 final report on administrative burden. The Committee reiterates the directives described in the Consolidated Appropriations Act, 2024 (Public Law 118–122).

The Good Science Project has been working on this issue for years. Imagine if DOGE had focused on improving government efficiency as to NIH-funded scientists rather than on increasing INefficiency to literally unheard-of levels.

Alternatives to Animal Research

The Senate report commends NIH for seeking alternatives to animal research:

New Approach Methodologies.—The Committee supports NIH Common Fund’s Complement Animal Research In Experimentation [Complement-ARIE] Program, intended to spur the development, standardization, validation, and use of new approach methodologies [NAMs] to more accurately model human biology. The Committee also encourages NIH, in new Announcements and other indications of funding opportunities, to continue consideration of NAMs as an option for areas of preclinical research when it is not appropriate to use human participants and where the use of NAMs has been demonstrated to support biomedical discoveries. The Committee further encourages NIH to collect and make publicly available a report that outlines how the use of vertebrate animal models in agency research contributes to the mission of NIH as well as efforts by the agency to encourage the use of new approach methods or strategies. This report should include examples of how other methods have been used in NIH research to reduce, replace, and refine the number of vertebrate animals used in research.

While we clearly can’t dispense with animal research entirely, there are many examples of such research that is basically useless in predicting what will work in humans. As one of many examples, Malcolm MacLeod once reviewed some 400 stroke treatments that purportedly worked in mice, but only ONE (tissue plasminogen activator, or TPA) worked in humans. We should actively look for alternatives.

Funding Younger Scientists

The Senate report says:

Expanding Support for Young Investigators.—NIH has been criticized for funding too many late career scientists while funding too few early career scientists with new ideas. The Committee is concerned that the average age of first-time R01 funded investigators remains 42 years old. More than twice as many R01 grants are awarded to investigators over 65 than to those under 36 years old. The Committee appreciates NIH’s efforts to provide support for early-career researchers through several dedicated initiatives, including the NIH Director’s New Innovator Award, Next Generation Researchers Initiative, Stephen Katz award, and the NIH Pathway to Independence Award. The Committee encourages NIH to continue supporting these important initiatives and to expand support for early career researchers by increasing the number of award recipients for these programs in future years. The Consolidated Appropriations Act, 2024 (Public Law 118–122) directed NIH to provide a ‘‘professional judgement’’ budget to the Committee to grow and retain the early career investigator pool, accelerate earlier research independence, and ensure the long term sustainability of the biomedical research enterprise. Building off of these efforts, the Committee directs NIH to provide an update on the activities to grow and retain early career investigators.

This all seems like a great idea. Middle-aged and elderly scientists have no monopoly on great ideas, and the opposite often seems to be the case (for example, the most innovative physics ideas in 1905 came from Einstein at age 26 or so).

Open Access to Scientific Papers

The Senate report says:

Article Processing Charges for NIH–Funded Research.—The Committee commends NIH for building on prior public access and datasharing reforms to initiate a process to address rising Article Processing Charges [APCs] that scientific journal publishers often charge NIH-supported scientists to publish the findings of their federally-funded research. NIH, and the scientists it funds, must be good stewards of taxpayer dollars and obtain as much research value as possible from limited resources. The Committee directs NIH to work with the scientific community to inform development of an APC allowable charge limit and, as part of that process, to: account for different publishing models, particularly U.S.based publishers focused on rigorous peer review and quality checks; support a robust American scientific research and publishing enterprise amid unprecedented global competition; and support the aims of gold standard science by establishing parameters to guard against potential abuses, including payment of APCs to journals that don’t prioritize research quality and integrity. Within 90 days of enactment, NIH is directed to brief the Committee on these efforts and its work to engage scientific journals on reasonable, sustainable APCs moving forward.

This is going to be interesting. Ever since people started demanding open access to published articles, journals (particularly the ones run by for-profit enterprises) have started to demand higher and higher fees for making articles openly available to read. The fee at Nature currently is around $12.6k! It is quite reasonable to ask whether NIH (or any scientific funder) should pay over $10k per article to make that article available to read.

Foreign Research Partners

The Senate report says:

Advancing Clinical Trials Through Subawards.—The Committee is concerned about the impact of the Administration’s policy to prohibit scientists from directing any funding to international research partners and the impact on clinical trials and human subjects research. Pediatric cancer, rare disease, HIV and infectious disease research rely on clinical trial participants and biospecimens from foreign countries in order to aggregate enough samples or patients for robust research. The Committee directs NIH to allow reimbursements and other funding arrangements with research partners abroad to foster pediatric cancer, rare disease, HIV and infectious disease research.

The Senate report seems unambiguously correct here. There are many cases where clinical trials simply can’t recruit enough participants in any single country.

For more than 80 years, economics has distinguished between microeconomics (how individual actors make decisions) and macroeconomics (how those decisions aggregate into large-scale patterns).1 This distinction has proven so valuable that we’ve forgotten how confusing economics was before it.

This micro/macro distinction enabled different methodologies, different types of evidence, and different policy levers. It revealed that what happens at the individual level can look very different when aggregated up to emergent, large-scale, societal patterns. Microeconomics and macroeconomics require different tools, ask different questions, and often reveal different truths.

Metascience needs the same clarity. Just as economics is about how incentives work in the marketplace for goods and services, metascience is about how incentives work in the marketplace for ideas and truth-seeking. And these incentives play out at the same micro- and macro-levels.

The Problem: Everything Is “Metascience”

Up to this point, “metascience” has been a broad and diffuse category, embracing everything from efforts to improve journal policies on preregistration, to large-scale analyses of citation patterns, to thought pieces on how NIH should change its grantmaking, to fraud detection and reproducibility studies, to ethnographies of labs, to launching new scientific organizations like Convergent Research or Speculative Technologies.

All of the above (and more!) has been lumped under one umbrella. This creates confusion about what metascience actually is, what methods it should use, and how different metascience efforts relate to each other. We’re trying to repair an epistemic economy without naming the fact that there is an economy, and that economies have both large-scale markets and individual minds.

Like financial economies, epistemic systems involve resource allocation (attention, funding, prestige), exchange mechanisms that create incentives (citations, collaborations), and trust (peer review, replication).

And like financial economies, individual rationality can produce collective irrationality. A scientist making the individually rational choice to avoid risky projects can contribute to a collectively irrational system where no one pursues breakthrough ideas.

To help clarify things, we should think of metascience at multiple levels, just like economics. If metascience were software, we’ve been trying to fix bugs in the user interface while ignoring the operating system (or vice versa).

The Distinction

Macro-metascience is about the political economy of funding, conducting, and publishing science at scale. In other words, it is about the large-scale policies and funding mechanisms that affect incentives, governance, institutions, etc. that make progress possible (or, in many cases, more difficult). This is the domain of the Good Science Project and other aligned organizations (such as the Institute for Progress or the Federation of American Scientists), that work on reforming federal agencies, redesigning grant mechanisms, and proposing journal reforms. Macro-metascience asks questions like: How should NIH structure peer review? What funding mechanisms best support high-risk research? How do citation patterns reveal bias versus creative innovation?

Micro-metascience, by contrast, is about individual scientists and the experience of discovery, i.e., how researchers actually experience reasoning, who they trust or doubt (and why), the mimetic pressures inside research groups, the ways in which they form conviction about evidence, and how they generate creative ideas. This is where Inês Hipólito’s work on cognitive and social epistemology and Omar Shehata’s work on mimetic engineering can shed light on how individual scientists navigate their intellectual environments.

Consider a concrete example of micro-metascience in action. If you’ve ever hung out at the hotel bar at a scientific conference, you will often hear some thoughts along the following lines: “Don’t quote me publicly, but no one really trusts such-and-such superstar in my field. We just can’t get that lab’s work to replicate.”

This gap between what scientists know privately and what they can say publicly is a micro phenomenon with macro consequences. If trust is created primarily through gossip rather than through publications, entire fields can be infected with unreliable methods and findings because no individual scientist will safely speak up to say that the emperor has no clothes. From the outside, the macro-level system may seem functional with lots of papers, citations, and grants, while the micro-level reality is epistemic dysfunction that only a few insiders actually know about.

The Distinction in Operation

This distinction reveals why so many well-intentioned metascience reforms can fail. They are aimed at only one level while ignoring the other level or merely assuming it will all go well.

As in economics, large-scale initiatives and reforms often ignore the cultural and psychological effects on individual scientists. You can redesign a national grant program, but if individual researchers inside the system are operating out of a mindset of fear, scarcity, or reputational risk, the reform may not work or could even backfire.2 Conversely, you can start a mass movement to encourage individual scientists to follow their curiosity, but if universities are less favorable towards early-stage exploration that “failed,” then individual scientists will behave as you would expect given their incentive to keep their jobs.

Take preregistration. At the macro level, preregistration at one time seemed like an obvious solution to publication bias and p-hacking, with idea being to require researchers to commit to hypotheses and analysis plans before seeing their data. Many journals and funding agencies have adopted preregistration requirements by this point.

But the micro level arguably tells a different story. If individual scientists are afraid that preregistration might increase their chance of a null result and thereby lower their chance of publication, they may instead try to game the system by preregistering every potential hypothesis and outcome.

Indeed, this is arguably what has happened in some fields: researchers submit vague preregistrations that allow maximum flexibility, or they preregister multiple analysis strategies and then selectively report the ones that “worked.” A macro-level intervention is therefore less effective than anyone anticipated because it didn’t account for how individual scientists would actually experience and respond to the new requirement.3

Similarly, let’s thinkg about national initiatives to sponsor so-called “high-risk, high-reward” research (a term that we dislike).4 At the macro level, agencies like NIH or NSF have created special funding programs specifically calling for such “high-risk” research. But at the micro level, if these programs are viewed cynically by individual researchers who submit work they have already completed (because that’s the only way to make it seem “safe” enough to get funded), then the initiative may not have its intended impact.

After all, researchers will respond rationally to their perception that “high-risk” is actually code for “we’ll still only fund things that look exciting but also highly likely to succeed.” We are aware of one scientist who took a close look at the NIH’s Pioneer awards (intended to sponsor ambitious scientists to take a “new scientific research direction”), and who found that most or all grantees were actually continuing the same research direction as before.

Perhaps the most powerful example is Katalin Karikó’s decades-long struggle to develop mRNA therapeutics. At the macro level, institutions repeatedly rejected her work: she was demoted and was told her research had no future. The macro-level system—with its emphasis on immediate results, conventional approaches, and established paradigms—couldn’t recognize the value of her exploration.

But at the micro level, Karikó’s individual persistence ultimately led to one of the most important medical breakthroughs of the 21st century (and a Nobel Prize). The interaction between these levels is crucial: whether a university or NIH will tolerate uncertainty will then affect an individual scientist is willing (or not) to persist with unpopular ideas. How many potential Karikós gave up because they lacked her combination of stubbornness, self-belief, and tolerance for professional humiliation?

The superstar economist Raj Chetty has written about “Lost Einsteins,” as in people who had the potential to be a scientific genius but never got the necessary education. Perhaps we also need to focus on “Lost Karikós,” as in scientific geniuses who did get the right education but were failed by the system.

Another example of how the two domains interact: A micro-metascientist might work with individual scientists to determine how disagreement affects someone’s willingness to undertake outside-the-box research, i.e., measuring things like psychological safety, the formation of trust (or not!), and willingness to share informal ideas. A macro-metascientist might redesign the NIH process to fund more outside-the-box research by changing how peer review ratings are scored, perhaps implementing variance-based selection where proposals with both very high and very low scores get funded.

These sorts of questions must be studied together, because the feedback loops between levels will determine what actually happens and whether an intervention works.

Comprehensive Metascience

A comprehensive theory of metascience would recognize that:

High-level policy, governance, and funding decide what even what counts as “science” in the first place. For example, the NIH’s emphasis on “significance” and “impact” forces researchers to frame exploratory work as if it already has clear applications. This requirement asks scientists to talk about supposed real-world applications before they are even able to understand the basic mechanisms at issue. The macro-level requirement thus shapes the micro-level experience of how scientists imagine their own work.

Local culture determines how individual scientists react to national policies, and in turn, whether their day-to-day behaviour is likely to lead to breakthroughs. When researchers internalize the message that only positive results are publishable or fundable, they will consciously or unconsciously avoid research ideas that might yield null results, even when those would be highly useful to the field. Even worse, they will be tempted to engage in questionable research practices (sometimes rising to the level of fraud). That gives us the worst of all worlds, in which people study marginal questions and then exaggerate the results (rather than studying big questions and telling the truth about their failures).

The interaction between levels creates emergent dynamics that can’t be predicted from either level alone. A scientist who is devoted to humility (a micro-level virtue) and the pursuit of breakthrough ideas might be employed by a series of universities that, for all practical purposes, punish negative results (a macro constraint). That scientist can experience cognitive dissonance between their desire to do great research and their desire to hold down a job. Maybe they leave science, maybe they become cynical, or maybe they find ways to pursue their curiosity in the shadows. In any event, we can’t understand the system if we only focus on one level at a time.

This piece is an attempt to understand the structure of the scientific system. If economics needed micro and macro to understand markets, science needs micro and macro to understand how to improve itself.

The Practical Implications

Recognizing the micro/macro distinction has practical implications for anyone who wants to improve science (including ourselves).

First, there is a gap that has caused many failures: nearly every metascience intervention operates at only one level. Reformers either redesign incentive structures from the top down while assuming scientists will respond as intended, or they train scientists in better practices while ignoring whether institutions will reward those practices. The micro/macro distinction makes this gap more explicit, and tells us why we should want our policies and interventions to address both levels simultaneously.

Second, we think we now have a more useful framework for understanding why reforms fail. When a seemingly well-designed macro-level intervention doesn’t produce the results we hoped for, we should ask questions like, “What is the individual experience of the scientists in the system? What are their fears, their incentives, and their sources of trust and creativity?” By asking those types of questions, the failure point would then become more visible.

Third, we are suggesting that effective interventions should ponder three questions ahead of time: 1. What are the macro-level constraints? 2. How will individual scientists experience these constraints? 3. What feedback loop operates between the two levels? Only when we can answer all three questions do we understand the system we’re trying to change.

This means, for instance, that designing a new funding mechanism requires both macro-level analysis (how does this change incentives? how does it interact with other funding sources? what behavior does it reward?) and micro-level understanding (how will scientists perceive this mechanism? what will it feel like to apply? who will feel comfortable using it versus who will self-select out? how will it affect scientists’ willingness to take risks?).

Conclusion: Designing for Both Levels

A new understanding of science and metascience is possible. Most importantly, good science doesn’t arise merely from having systems that look the best from a top-down perspective. The most well-designed funding mechanism may run aground if the local culture is one of fear, scarcity, and ferocious competition for grants. The most brilliant peer review process can’t manufacture individual scientists who have faith in their own judgment and are willing to pursue unpopular ideas.

Ultimately, good science is produced by individual people and small teams that are all embedded within larger systems. And if we want better science, we have to design for both levels, the micro and the macro. We need to understand how policy shapes personal experience, and how personal experience in turn affects which policies are feasible.

The micro/macro distinction gives us the language to do this. It’s time metascience became as sophisticated about its own structure as economics has been about markets. We have the macro tools, such as policy reform, new funding mechanisms, new institutions, and more. We’re developing the micro tools: understanding how trust and psychological safety emerge, and how creative impulses are given the freedom to exist.

We need to work on both levels. Only then can we build the epistemic economy we need one that, produces not just more science, but better science, breakthrough science, science that actually helps us understand the world.

• In Asterisk Magazine, Karthik Tadepalli makes the case that “Ideas Aren’t Getting Harder to Find.” It’s based on recent economic literature (such as this Census Bureau paper) suggesting that even though we spend far more on science and R&D than we did decades ago, the bottleneck to increased productivity and economic growth is less about the difficulty of discovering new scientific ideas, and more about our downstream capacity to bring them to market. Not sure if this is good news or bad news (given the difficulty in reforming some of those downstream processes).

• Science magazine obtained an internal NSF memo announcing that NSF is lowering its internal requirements as to peer review (or, as they call it, merit review). In brief: “The changes permit as few as one outside review rather than the current minimum of three, end the routine use of expert panels to discuss those individual reviews, and give program managers greater authority to recommend which proposals should or should not be funded.”

  • A number of folks seem upset at the prospect, but I’m cautiously optimistic (the devil, as always, is in the details). The thing is, despite its widespread use, there really isn’t any empirical evidence for peer review for scientific grantmaking! A RAND report found as much in 2018 (and things haven’t changed since then):

    • “Judging whether peer review is demonstrably better than any other system is impossible because of the lack of comparators. No funding agencies have made significant use of other allocation systems. Even comparisons between or research on peer review systems is limited, with most studies examining the peer review process of one particular funder in one particular context, and few go beyond process measures to judge improvement.”

  • DARPA functions at a high level without rigid peer review requirements for every specific proposal. In principle, NSF program officers ought to be able to do likewise, i.e., to make informed decisions based on their own expertise. It would be preferable, of course, if NSF could roll out such a policy on a randomly-staggered basis so as to measure the effects . . .

• A report came out this week on reproducibility problems both in the crystallography literature and some major databases, with some calling for a Google paper to be retracted. Not my field, but interesting to see all the drama.

• I enjoyed this piece by Adam Marblestone, Anastasia Gamick, and Joseph Fridman on lessons learned from the first 5 years of Focused Research Organizations. They usefully clarify the places where FROs are useful, and where they are not.

• Just came across an NBER working paper from July 2025, by Amitabh Chandra and Connie Xu. It’s called “Where Discovery Happens: Research Institutions and Fundamental Knowledge in the Life Sciences.” Their sample includes 560,000 articles in the life sciences written by 37,809 scientists who switched institutions during the time period at issue (this is important, because it helps to control for the underlying productivity, reputation, connections, etc. that any given researcher might have). I’m not going to get into the weeds of exactly how they define “productivity” (it includes number of papers, authorship position, weighting for the journal’s impact factor and the paper’s own citation count, and more).

  • Bottom line: “Between 50-60% of a scientist’s research output is attributable to the institution where they work, and two-thirds of this effect is driven by the presence of star researchers.”

  • A policy implication: “making public or philanthropic funding less generous at institutions that have high per-scientist output will directly reduce the production of knowledge, especially commercially relevant knowledge.”

• Going back to at least the late 1990s, the NIH had a policy (here’s the 1998 version) requiring that if you ask for more than $500,000 per year in direct costs, you have to get advance permission from NIH staff that they will agree even to look at the application. The rationale at the time was that such large awards “are difficult to manage” and plan for within the usual budgetary process, so NIH wanted a heads up — and the right to refuse to accept such proposals at all. As of Dec. 3, 2025, that decades-old policy has been rescinded. No more need for advance permission.

  • Several years ago, an NIH leader speculated to me that since the $500,000 limit hadn’t been adjusted for inflation, it could have the unintended consequence of incentivizing researchers to propose smaller and smaller studies over time, which would be particularly negative as to clinical trials (which can be expensive to run properly).

  • It is odd to me that any government agency has policies with specific dollar amounts, and doesn’t adjust for inflation over a 27-year time period. $500,000 in 1998 would be worth $999,000 today. Seems like the policy was long overdue for a serious amendment. And ditching the policy seems sensible—it was a lot of red tape that doesn’t really seem necessary.

• This is going to be a little more geeky than usual (!), but a long-standing pet peeve of mine is seeing journalists and even scientists report the “variance explained” by some variable as if it represents the possible causal impact of that variable. E.g., someone will say, “school spending explains only 5% of the variance in student outcomes,” as if that shows that school spending doesn’t matter (or even as if we could spend zero and get similar outcomes).

  • That is a completely wrong view of “explained variance.” Think of it this way: Imagine that no one in the world has a birth defect that results in anything other than two legs, and all of the variance in “number of legs” is explained by “accidents and disease.” That does not mean that genes have no causal impact as to “number of legs”—it is only because of genes that we all (or mostly all) have two legs in the first place. The causal importance simply isn’t the same thing as “variance explained,” which crucially depends on the underlying amount of variance at issue.

  • A couple of economists have a recent working paper called, “The Explanatory Power of Causal Effects.” I don’t know these economists myself, but they seem to have gotten comments from a bunch of folks that I do know and respect (Raj Chetty, Peter Hull, Kosuke Imai, Larry Katz, and Sendhil Mullainathan, among others). Their motivation is that while economists know that R-squared isn’t causal, “to our knowledge, there is no measure of the variation in an outcome causally explained by a variable.” They propose a new measure that they call Causal R-squared, or CR². It requires both 1) an experiment that shows the effect of X on Y when randomly assigned, and 2) an observational dataset showing the relative variances of Y and X in the population. The point is to estimate how much an intervention might causally affect the variance of Y conditional on X.

  • They apply this new measure to several specific questions. For example, they “assess the share of variation in blood pressure explained by sodium intake: sodium causally explains 7% of the variation in men’s blood pressure, but less than 1% in women, despite similar causal effects. The gender difference arises because women’s blood pressure varies more for reasons unrelated to sodium.”

  • They also point out who will be interested in this and why: “Normatively, a policy‐maker may care about X, even though it explains little variation in Y. Goldberger (1979), writing in the context of genetic heritability, gives the example of eyesight: population variance in eyesight is largely genetic, but there is still great value in prescribing glasses. For this reason, CR² is more relevant to a scientist seeking to understand the sources of naturally‐occurring variation in Y, than to a policy‐maker seeking to affect the value of Y.”

  • For more, see this Twitter thread by one of the authors.

• On my watchlist: “The Thinking Game,” a full-length documentary on DeepMind. Looks amazing.

At the Good Science Project, we have often written about the need for institutional diversity in science, as opposed to the current institutional isomorphism (where most organizations look and operate the same way). As I said in an interview published yesterday:

When all institutions look the same, their output tends to be the same as well. Tens of thousands of scientists (usually government-funded) labor within the same constraints, incentives, and pressures to conform.

If we want to see a higher rate of scientific breakthroughs, we should pursue institutional diversity for that reason alone. Breakthroughs are by definition a break with the status quo in some respect — and if the status quo is too uniform and powerful everywhere, it would be hard for any individual scientist to contradict the entire ecosystem that provides his or her entire livelihood and professional reputation.

Institutional diversity may be one of the key ways to produce ideological diversity in science — that is, scientists who are empowered to take a different view precisely because they have a different position and source of salary, rather than being forever hobbled by those constraints.

With all of that in mind, these are exciting times.

First, just today, the National Science Foundation’s Technology, Innovation and Partnerships directorate at long last announced its Tech Labs initiative, which is intended to provide $10-$50 million a year to independent research teams (and yes, that is a per team dollar amount, not the initiative’s entire budget).

The intent is to provide “entrepreneurial teams of proven scientists the freedom and flexibility to pursue breakthrough science at breakneck speed, without needing to frequently stop and apply for additional grant funding with each new idea or development.”

The idea has many precursors, including all of the independent research labs and organizations going back several decades, the recent burst of philanthropy for new institutes and organizations, the idea of focused research organizations (here’s a good piece from today), Caleb Watney’s excellent piece proposing X-Labs, and Jeffrey Tsao’s proposal for Bell Labs X.

But this is the first time the federal government has gotten into the business of actively pushing for institutional diversity and for scientific funding at the team level.

Huge, if it works.

And there is reason to think it might work. Rather than relying on the often-invoked Bell Labs and Xerox PARC, let’s look at other organizations — symphony orchestras and professional sports teams.

The New York Philharmonic doesn’t make its principal viola player spend all of her time trying to find individual grants to support her salary while writing up extensive reports on how much she practiced every month and an analysis of her part in each performance. Instead, the orchestra raises money for the institution from donations, ticket sales, and album sales. The institution is then responsible for selecting the best performers and enabling them to do their best work as a team.

Similarly, the Golden State Warriors don’t make Steph Curry spend all his time writing up 50-page proposals for each $1 million chunk of his salary, along with regular written reports on his performance in each of the 82 games per season. Instead, they let him focus on practicing and playing basketball. Indeed, they even hire a shooting coach to develop unique drills so that Curry, already the best shooter in world history, can continue to get even better.

High-performing institutions usually work together as a whole, rather than making their individual employees spend so much time hustling for money. We should probably be doing more of this in science. In fact, I can imagine making a case for block funding to many universities and academic hospitals, which are (and will remain) an important cornerstone of scientific research.

Also worth noting: the press release says the funding is for “research teams outside of traditional academic institutions,” but there’s a bit of nuance to that. If you read the full PDF linked on this page, the teams that are eligible include “teams from within academia or industry that are ready to scale a technology or vision beyond their existing institutional structure(s).”

As well, I’m happy that NSF decided to include this as one of the 5 strategic priorities for each Tech Lab:

This is reminiscent of Jeff Tsao’s point that if we want to have a modern version of Bell Labs, we need to recreate a couple of conditions that made Bell Labs possible: (1) Alignment with a major industry focused on the frontier of applied R&D problems, but (2) the intellectual freedom to pursue curiosity-driven research and to pivot towards surprising findings.

Tsao argues that in today’s market, the intellectual freedom in (2) will arise only if government/philanthropy supports a research organization, while to attain (1), the organization needs to be deeply embedded with a major corporation that regularly surfaces cutting-edge problems that actually matter to the real world. Thus, we need public/private partnerships here. [NSF’s team was well aware of Tsao’s ideas.]

Finally, as I told the NSF team a few weeks ago, we need to consider the following question: “What is the optimal rate of failure that we’re looking for?” That is, given the desire to pursue “ambitious goals,” if all of the grantees come back a year or two later and say, “We succeeded in hitting 100% of our milestones!,” then someone wasn’t ambitious enough.

Nothing is guaranteed in life, and certainly not in science. If a scientific portfolio has a 100% success rate, then it is probably so incremental and obvious as to be worthless.

But what’s the optimal failure rate? (That is, failure not for reasons of incompetence or mismanagement, but because the scientific question can’t be solved quite yet.)

Well, that depends. The optimal failure rate has to be determined by anyone running a scientific funding program in advance—does your program want a 10% failure rate, 50%, 90%, or what? That expectation directly changes the amount of actual risk that a program will undertake.

Even at DARPA, it can be difficult to acknowledge failure (as Adam Russell has pointed out). Whether at NSF or anywhere else, anyone who wants to do/fund ambitious science has to get more comfortable with publicly acknowledging what didn’t work. Discussing failure has to be more normalized.

Anyway, NSF is asking for public comments to be submitted by Jan. 20, 2026. Send in your thoughts!

***

Second, just yesterday, the UK government signed an agreement with Google’s DeepMind (inventor of AlphaFold). As per DeepMind’s post, the initiative will: 1) modernize government by creating new AI tools for public servants, 2) develop new programs to improve education, and most significantly for our purposes, 3) “establish Google DeepMind’s first automated laboratory in the UK in 2026, specifically focused on materials science research.”

The rationale:

By directing world-class robotics to synthesize and characterize hundreds of materials per day, the team intends to significantly shorten the timeline for identifying transformative new materials.

Discovering new materials is one of the most important pursuits in science, offering the potential to reduce costs and enable entirely new technologies. For example, superconductors that operate at ambient temperature and pressure could allow for low cost medical imaging and reduce power loss in electrical grids. Other novel materials could help us tackle critical energy challenges by unlocking advanced batteries, next-generation solar cells and more efficient computer chips.

As with the NSF initiative, it is good to see a government agency partnering with a fairly new scientific organization to give funding at the institutional level to pursue ambitious goals. And given DeepMind’s track record thus far, I expect great things.

***

Third, Eric Gilliam (currently with Renaissance Philanthropy, while remaining a Good Science Project fellow) has been doing exciting work extending the ideas he developed over the past few years about creating new organizations along the lines of BBN (the ARPA contractor that actually developed the first version of the Internet).

Thanks to Renaissance Philanthropy’s engagement with the British agency ARIA, Eric’s work led to the Dec. 1 announcement of what they’re calling the Frontier Research Contractor Launchpad.

The goal is to build “a new class of ambitious, applied R&D organisations in the UK,” and a list of the initial organizations can be found here.

Eric has a bigger idea to fund more such organizations across the ecosystem, so get in touch if you’re interested.

***

PS: I used em-dashes several times above, as I usually do (and have done for many years). Just realized that given the recent trend of people claiming to identify AI-assisted writing, it might be worth clarifying: Everything above was written solely by me in the past 3 hours, and I use em-dashes because I grew up reading great writers who used them all the time: Dickens, Austen, Dostoevsky, Louisa May Alcott, Mark Twain, Herman Melville, etc. It is ironic that just because AI models were evidently trained on classic writers, people who aren’t well-read seem to think that AI models invented the em-dash! I will never stop using em-dashes.