This is extremely cool. I admit to having a naive understanding of the subject, but that contextualizes my also-naive question that nevertheless I think is operative when it comes to methodology designs like this: how confident are you that you're managing to avoid overfitting and how (as time progresses) are you updating your conceptions of fit in response to new information without likewise deepening the degree of overfit?
Good question, not naive. The paper includes a lot of data that I didn’t elaborate on here. Most relevant to this issue, we actually tested our model over two different, dispersed time frames, 1994-1997 and 2014-2017; it’s worth noting that we were very careful about truncating historical data, so that at no time was there information leakage from the future. We found for these two time periods twenty years apart, the kinetics of signal formation are the same. This suggests that our model is reasonably durable to the passage of time, although of course it’s always important to remain alert to the possibility of drift.
Many of these later-period predictions are also starting to bear fruit; for example, we called optical coherence tomography angiography (OCT-A) a breakthrough based on a signal in 2016 data. Fujimoto, Huang, and Swanson were awarded a Lasker in 2023 for developing the technique. We also predicted the use of mobile phones to deliver healthcare (think teledoc appointments) based on a 2015 signal.
Ultimately time will tell us how we’ve performed, and I’m looking forward to seeing how these later predictions continue to unfold over the next few years.
*nodnodnod*. That makes sense. Basically "we're not taking the sweep of the arc of the past as one unified thing; we're giving ourselves multiple independent opportunities to fit and testing whether the underlying mechanics that seem to generate this fit make sense and are consistent", or something close to that anyway :)
It’s also important to ask whether the signal can be manipulated. This seems moderately robust, but any signal that gets used for allocating funding will then be a target for manipulation, so you don’t want to just encourage people to generate the signal that you’re following.
Always necessary to think about gaming. We have thought very carefully about how this approach might be gamed, and have, to the extent possible, made design choices to mitigate against it. Most importantly, because of the underlying math, it's quite difficult to cast enough "votes" for yourself to trigger the signal. No matter how well designed the measure, though, it's important to keep humans in the loop for orthogonal validation.
This is important work, and I’ll say up front I’m on the side of the researchers here — I write as a research admin and as a mother who wants a better world for my kids and, so, for all kids. The Karikó pattern is real and infuriating, and getting good science backed before it’s five years too late is worth chasing. So I’m not arguing against the model. I’m curious what it optimizes for.
My concern is the signal's thinness. A model tuned to citation bursts rewards fields that already have people with the resources to flock to them — WEIRD, well-funded, mostly elite institutions generate that flurry; the ecosystems I volunteer in can’t, even when they’re solving everyday problems with huge returns for who actually gets to use science. So “merit” risks becoming a faster version of patronage. (That’s a throughline found in recent work by Musa al-Gharbi, Raj Chetty, and lately Ada Palmer’s Inventing the Renaissance — and the legitimate/illegitimate line is an old, engineered one, going back to the 12th-century Church deciding who could inherit, but found in examples throughout today and history touching on language, ethnicity, linguistics, religion, culture, gender, etc.)
A non-WEIRD scientist I worked with called this the “Marvel Effect”: elites hoard the resources, inequality compounds the problems, and then someone swoops in with an Iron Man fix and expects the hoarding that produced it to be forgiven because they’ve arrived to save the day — from a problem the system that funded them arguably helped create to their (albeit not intentional) benefit.
There’s also a distribution question your piece raised for me. You’d run the experiment through active program managers with authority to recruit — the incumbent apparatus. Al-Gharbi estimates that of every dollar aimed at redistribution, roughly 75 cents is absorbed by the institutions doing the redistributing before it reaches anyone. I’d genuinely want to know how a breakthrough fund avoids the same skim.
And this is where your call for “a framework that doesn’t repackage prestige” might go further. Economic complexity is the closest thing we have — the renowned César Hidalgo’s ECI outpredicts schooling and capital metrics because it measures latent, path-dependent capability, which is really your seed-corn point made rigorous. But even ECI has no column for care: it rewards what’s scarce and recombinable, so the abundant, unrecombinable work that keeps an ecosystem alive scores near zero. A citation model is thinner still. Researcher Angus Fletcher’s primal intelligence names what these metrics miss: the intuition, imagination, emotion, and common sense that actually move a field, which live in the very people a signal strips away.
So I’d love your read: is there a way to fund, as Fletcher puts it, the probable while protecting the possible — and to build a signal that can see the people the metric tends to erase? That, to me, is the difference between making our own luck and deciding what luck means for everyone else.
It's critically important to preserve the possible.
Our approach identifies probable breakthroughs, but as I say in the piece, these topics don’t come from nowhere. They often seem to arise when two different fields converge, or when an existing field splits. We can’t say exactly why these merges and splits happen, but we can be certain that serendipity and pursuing curiosity (which is another way to say pursuing intuition and imagination) are essential factors.
How you go about protecting that space gets into how you manage the (much larger) proportion of your portfolio, as an agency or philanthropy, that supports what Kuhn calls “normal science”; and that is a complex subject that I think deserves an entire essay of its own.
This is extremely cool. I admit to having a naive understanding of the subject, but that contextualizes my also-naive question that nevertheless I think is operative when it comes to methodology designs like this: how confident are you that you're managing to avoid overfitting and how (as time progresses) are you updating your conceptions of fit in response to new information without likewise deepening the degree of overfit?
Good question, not naive. The paper includes a lot of data that I didn’t elaborate on here. Most relevant to this issue, we actually tested our model over two different, dispersed time frames, 1994-1997 and 2014-2017; it’s worth noting that we were very careful about truncating historical data, so that at no time was there information leakage from the future. We found for these two time periods twenty years apart, the kinetics of signal formation are the same. This suggests that our model is reasonably durable to the passage of time, although of course it’s always important to remain alert to the possibility of drift.
Many of these later-period predictions are also starting to bear fruit; for example, we called optical coherence tomography angiography (OCT-A) a breakthrough based on a signal in 2016 data. Fujimoto, Huang, and Swanson were awarded a Lasker in 2023 for developing the technique. We also predicted the use of mobile phones to deliver healthcare (think teledoc appointments) based on a 2015 signal.
Ultimately time will tell us how we’ve performed, and I’m looking forward to seeing how these later predictions continue to unfold over the next few years.
*nodnodnod*. That makes sense. Basically "we're not taking the sweep of the arc of the past as one unified thing; we're giving ourselves multiple independent opportunities to fit and testing whether the underlying mechanics that seem to generate this fit make sense and are consistent", or something close to that anyway :)
Yes!
It’s also important to ask whether the signal can be manipulated. This seems moderately robust, but any signal that gets used for allocating funding will then be a target for manipulation, so you don’t want to just encourage people to generate the signal that you’re following.
Always necessary to think about gaming. We have thought very carefully about how this approach might be gamed, and have, to the extent possible, made design choices to mitigate against it. Most importantly, because of the underlying math, it's quite difficult to cast enough "votes" for yourself to trigger the signal. No matter how well designed the measure, though, it's important to keep humans in the loop for orthogonal validation.
This is important work, and I’ll say up front I’m on the side of the researchers here — I write as a research admin and as a mother who wants a better world for my kids and, so, for all kids. The Karikó pattern is real and infuriating, and getting good science backed before it’s five years too late is worth chasing. So I’m not arguing against the model. I’m curious what it optimizes for.
My concern is the signal's thinness. A model tuned to citation bursts rewards fields that already have people with the resources to flock to them — WEIRD, well-funded, mostly elite institutions generate that flurry; the ecosystems I volunteer in can’t, even when they’re solving everyday problems with huge returns for who actually gets to use science. So “merit” risks becoming a faster version of patronage. (That’s a throughline found in recent work by Musa al-Gharbi, Raj Chetty, and lately Ada Palmer’s Inventing the Renaissance — and the legitimate/illegitimate line is an old, engineered one, going back to the 12th-century Church deciding who could inherit, but found in examples throughout today and history touching on language, ethnicity, linguistics, religion, culture, gender, etc.)
A non-WEIRD scientist I worked with called this the “Marvel Effect”: elites hoard the resources, inequality compounds the problems, and then someone swoops in with an Iron Man fix and expects the hoarding that produced it to be forgiven because they’ve arrived to save the day — from a problem the system that funded them arguably helped create to their (albeit not intentional) benefit.
There’s also a distribution question your piece raised for me. You’d run the experiment through active program managers with authority to recruit — the incumbent apparatus. Al-Gharbi estimates that of every dollar aimed at redistribution, roughly 75 cents is absorbed by the institutions doing the redistributing before it reaches anyone. I’d genuinely want to know how a breakthrough fund avoids the same skim.
And this is where your call for “a framework that doesn’t repackage prestige” might go further. Economic complexity is the closest thing we have — the renowned César Hidalgo’s ECI outpredicts schooling and capital metrics because it measures latent, path-dependent capability, which is really your seed-corn point made rigorous. But even ECI has no column for care: it rewards what’s scarce and recombinable, so the abundant, unrecombinable work that keeps an ecosystem alive scores near zero. A citation model is thinner still. Researcher Angus Fletcher’s primal intelligence names what these metrics miss: the intuition, imagination, emotion, and common sense that actually move a field, which live in the very people a signal strips away.
So I’d love your read: is there a way to fund, as Fletcher puts it, the probable while protecting the possible — and to build a signal that can see the people the metric tends to erase? That, to me, is the difference between making our own luck and deciding what luck means for everyone else.
It's critically important to preserve the possible.
Our approach identifies probable breakthroughs, but as I say in the piece, these topics don’t come from nowhere. They often seem to arise when two different fields converge, or when an existing field splits. We can’t say exactly why these merges and splits happen, but we can be certain that serendipity and pursuing curiosity (which is another way to say pursuing intuition and imagination) are essential factors.
How you go about protecting that space gets into how you manage the (much larger) proportion of your portfolio, as an agency or philanthropy, that supports what Kuhn calls “normal science”; and that is a complex subject that I think deserves an entire essay of its own.