Making Our Own Luck
What if we could predict transformative scientific breakthroughs before they happen?
We are cross-posting a piece that former NIH official Kris Willis wrote for Macroscience. Kris is now the President and Founder of the Woodley Park Institute, for which I’m a board member.
Picking winners or hedging bets?
Federal science agencies have long wrestled with the question of how to ensure their billions of dollars of grants and contracts result in maximum benefits for Americans.
On the surface, it might seem that a goal-oriented, interventionist style of management would be the best policy: prioritize our most pressing problems and distribute funding accordingly.This top-down strategy creates a unique hazard, though: bad choices can send resources in the wrong direction, resulting in slow progress that impedes the delivery of tangible benefits to citizens.
Moreover, if decision makers pursue an applied advance when the underlying scientific principles remain poorly understood, they risk costly failures and a loss of support for the research enterprise at large. Examples include Nixon’s 1971 War on Cancer, undertaken on the premise that a cure could be achieved in less than a decade, the drive to build an operational commercial fusion plant by the mid-1990s, and the National Plan to Address Alzheimer’s Disease, which failed to produce the effective treatments and prevention strategies promised by 2025.
The contrasting laissez-faire approach is to hedge your bets. If the path to an advance is impossible to predict, then the wisest choice might be to distribute funding to as widely as possible. The main risk of this strategy is that spreading funding thinly may limit the resources available to the most promising research, resulting in the same negative outcomes as the interventionist approach: limited support for the most promising topics, slow progress, and delayed returns.
In reality, at science agencies like the National Science Foundation and National Institutes of Health, program managers navigate between these two extremes, setting some funds aside to target high-priority areas and using the rest to cover the widest possible range of meritorious proposals. This middle path seems like the most rational way to proceed, but it still requires making choices. How much should be set aside for high priority topics? What should those topics be? How should we define merit? Expert opinion can provide a helpful guide, but leading experts can have remarkably different opinions about what matters and what should be prioritized. Moreover, experts and administrators alike can be risk averse, giving an edge to high-profile, well-established concepts.
What’s really needed is a data-driven framework to help guide decision-making, one that doesn’t simply repackage prestige or incumbency.
Recognizing breakthroughs years in advance
Many breakthroughs fail to attract funding or recognition at the earliest stages of development. Katalin Karikó, for example, spent years struggling to win grants for work on mRNA that would eventually win her and Drew Weissman a Nobel Prize. Similar stories can be told about Jim Allison’s groundbreaking studies leading to the development of cancer immunotherapy, Stan Pruisner’s demonstration that prions can self-replicate without DNA, or Robin Warren and Barry Marshall’s proof that Helicobacter pylori, not stress, causes ulcers.
Yet at some point, the scientific community recognizes and responds to transformative breakthroughs like these. How does that change happen? And what if science funders could identify the areas that are on the cusp of a breakthrough?
Those questions led my collaborators and me to study how scientists responded to past breakthroughs. Based on patterns in that data, we developed a means of predicting future breakthroughs, using biomedicine as a proof of concept.
It’s worth taking a moment to describe how our approach works. Briefly, our prediction process begins by grouping the 18 million or so peer-reviewed papers in PubMed, the authoritative database of biomedical research, into roughly 50,000 unique topics. A small subset of these topics are so widely acclaimed as breakthroughs that they have been recognized with a major prize like a Nobel or a Lasker Award. We theorized that studying the development of these outliers over time, especially before they gained acclaim, was the key to identifying future instances of them.
After defining topics, we chose 21 representative breakthroughs and asked what they looked like before they won prestigious prizes. To answer that question, we re-ran our topic-mapping algorithm repeatedly, winding back the clock one year each time, stringing together the years to follow the progress of each field as it grew and advanced.
We found that in each case, the years leading up to a discovery followed a pattern: a burst of papers on a new, rapidly evolving topic, many of which quickly became influential. These characteristics—the percent of papers that are new, the percentage of papers that are brought in from other topics, and the influence of each individual paper—can all be measured separately for any given topic. Together, they act as a predictive signal that is detectable an average of five years before a transformative discovery is made, and up to thirty years before the discovery receives a major prize. Scanning the current research landscape for topics that display this signal allows us to predict what work will likely produce a future breakthrough. Each year we analyzed includes four or five signals, a number that appears to have remained constant across a twenty year time frame.
In their classic work on the sociology of science, Bruno Latour and Steve Woolgar established that the desire to participate in a major discovery is an important motivator for scientists as they consider whether they should take up (or abandon) a research problem. Although further work is needed, the simplest explanation for our results is that this behavior is widespread enough to detect at scale. A foundational advance, or something like it, draws the attention of scientists and causes them to change the direction of their research. They vote with their feet, staking their own reputation and careers by publishing on and citing the new idea. That upswell of interest is the foundation of our breakthrough signal.
What kind of research does our model identify as a breakthrough?
After building a model based on the patterns in our 21 preselected breakthroughs, we tested it by identifying all early signals of discovery from 1994 through 1997. Of the 18 signals we found in that time period, 17 can be traced forward in time to a breakthrough. Examples include super-resolution fluorescence microscopy (see the image above), the directed evolution of proteins and enzymes (Chemistry Nobel, 2018), and sequencing the human genome (National Medal of Science, 2008).
The discovery of the role of leptin signaling in obesity is a particularly interesting case study. Around 1950, scientists noticed the existence of a type of lab mouse that suffered from a unique inherited form of obesity. When given the freedom to choose their own diet, these mice ate more than normal and preferred food that was high in fat. Researchers had few clues as to why the animals overate; early experiments pointed to the existence of a soluble ob factor found circulating in the bloodstream that regulated appetite, but its identity was unknown.
In 1994, Jeffrey Friedman and colleagues identified and sequenced the ob gene, noting that the protein it encoded appeared to fit the profile of a circulating factor. A year later, a trio of high-profile publications demonstrated that ob was a hormone that regulated body weight and fat deposition by regulating appetite. One of these was co-authored by Friedman, who christened it leptin from the ancient Greek word for thin. Using our model, the data from 1996 make it clear this advance would become a breakthrough. Scientists had flocked to study the biology of the new hormone, generating the flurry of new papers and citations required to produce a signal. One of these demonstrated that serum leptin concentrations reflected the amount of adipose tissue in the human body, providing clinical validation of the earlier mouse studies. Before the identification of leptin, scientists were unable to point to a specific molecule that controlled appetite and adiposity, so that it was impossible to rule out non-physiological causes of obesity; its characterization fundamentally changed our concept of weight gain. In 2010, Friedman and fellow pioneer Douglas Coleman were awarded a Lasker for the discovery.
Our work also identifies breakthroughs in clinical practice and behavioral and social sciences, even though these areas are less likely to draw the attention of major prize committees. One example that falls into this category is the introduction of endovascular aneurysm repair, a minimally invasive surgical procedure for the treatment of abdominal aortic aneurysm, a leading cause of death among Americans over the age of 65. Relative to the previous standard of care, it reduces in-hospital mortality by almost four-fold. In 2017, Timothy Chuter received the American College of Surgeons’ prestigious Jacobsen Innovation award for his multiple refinements to the technique.
Another under-recognized breakthrough identified by our model is the development of standardized survey instruments to assess the quality of life for HIV+ patients. The antiretroviral cocktails introduced in the mid-1990s reduced AIDS mortality, but treatment came with serious side effects. The new evaluations showed that patients whose disease progressed had worse physical functioning than Americans with other chronic diseases, while those who were asymptomatic maintained a health-related quality of life on par with the overall US population. This evidence made it possible for patients and activists to argue that even in the absence of a total cure, and in spite of the side effects, alleviation of symptoms was a worthwhile priority for clinicians. My colleagues and I have failed to identify any significant recognition for the physicians and health policy experts who pioneered this advance, although in 2016, researchers called for the World Health Organization to set targets for good health-related quality of life as part of its framework to end the AIDS pandemic.
From forecasting to funding
How should science funders use our predictions?
Any serious discussion of what we should do with this new capability needs to begin with an understanding of its limitations. We can’t (and shouldn’t) invest all of our resources in the small number of topics predicted to produce breakthroughs. Innovative new fields are born from existing ones; they rarely arise de novo. If funders don’t maintain a diverse portfolio, the breakthroughs of tomorrow will have no antecedents. Think of this as eating the seed corn, or killing the goose that lays the golden egg.
There are also multiple goals of funding beyond basic scientific discovery, including translating discoveries into clinical practice, developing new technologies, supporting economic development, and training new scientists. The investments needed to accomplish these goals are likely different from those required to support emerging breakthroughs.
Finally, we must be aware that we may not capture every instance of a breakthrough. The concept is fuzzy, and not every significant discovery is recognized with a major award. Further, breakthroughs are rare, meaning that our dataset is, by necessity, small. This makes formal estimates of accuracy challenging.
Although our overall success rate appears to be high, any one signal may turn out to be a false positive. We found one apparent example: the development of the third generation COX-2 inhibitors, Vioxx and Celebrex, which were much celebrated in the early 2000s as effective but non-addictive pain relievers, meets all the algorithmic criteria of a breakthrough. Five years after it received FDA approval, researchers demonstrated that Vioxx was associated with serious cardiovascular complications, and the manufacturer voluntarily pulled the drug from the market. Celebrex remains available, although like other NSAIDs, it carries an FDA warning for increased cardiovascular risk. Decision makers who were overly focused on the breakthrough potential of this research might have wasted substantial resources nurturing an area that ultimately proved to be a failure—but so would those who relied on expert opinion at the time.
That said, the potential benefits of funding the next mRNA vaccine or cancer immunotherapy years earlier than we might have are large. When we can’t assume the outcome of an action, it’s worth conducting an experiment. Can targeted support for areas that are poised to produce a breakthrough increase the return on scientific investment without cannibalizing future advances?
With the necessary administrative infrastructure in place, conducting this experiment would be straightforward. First, identify all the breakthrough signals between 2018 and 2023, select half at random, and commit to a decade of investment. For biomedicine, a six year window should yield about 30 topics with the potential to produce a breakthrough. Providing strong support to half of these might be accomplished for $300 million a year. That’s roughly half of the recent annual budget of the NIH Common Fund or around 20% of the fiscal year 2025 budget for ARPA-H. After five years, then again at ten, compare to see which group — the one that was actively managed, or the control — produced more breakthroughs, and on what timescale. Although we used biomedicine as an example, the same experiment could just as well be run for physics, materials science, or any other discipline.
Any attempt to undertake such an experiment should keep three points top of mind. First, we can predict the topic of future breakthroughs, but the people and vision for how to move the work forward are still important. In practice, this means success requires active program managers with the expertise to develop the science and the authority to do so, including by recruiting investigators. Second, we would be wise to learn from the failures of past attempts to fund transformative research. High among these are funding modestly repackaged work, confusing the genuinely novel with the merely unfamiliar, and relying too much on incumbent investigators. All of these risks can be mitigated by thoroughly and intelligently integrating high quality data into portfolio management. Finally, we need to have patience. Even with strong funding support, producing a breakthrough takes time.
If supporting breakthroughs is the best way to nurture the birth of new fields, we also need to accurately describe and properly manage the other two stages, so that we invest wisely throughout the productive lifetime of ideas and divest once they reach the point of diminishing returns. We would benefit from more study of all of these phases, especially the transition from one to another. More research could answer questions like:
When does a breakthrough become an established field?
How do the availability of funding, the size and characteristics of the available workforce, and the accumulation of evidence that disagrees with prevailing models affect a field’s decline?
Exactly how do old ideas give birth to new ones, and can we speed up the process?
The more we understand about the dynamics of scientific progress, the better we can maximize the odds that our curiosity leads to results. I’d call that finding a way to make our own luck.
 | A guest post by
|
This is extremely cool. I admit to having a naive understanding of the subject, but that contextualizes my also-naive question that nevertheless I think is operative when it comes to methodology designs like this: how confident are you that you're managing to avoid overfitting and how (as time progresses) are you updating your conceptions of fit in response to new information without likewise deepening the degree of overfit?
It’s also important to ask whether the signal can be manipulated. This seems moderately robust, but any signal that gets used for allocating funding will then be a target for manipulation, so you don’t want to just encourage people to generate the signal that you’re following.