yes yes yes. if we take it to the abstract, I think te the job of making meaning out of all this information we have is one of the most important things people can do today
I have given this some thought as well, and from my experience methodological reviews get significantly better with field-specific specs. I think these field method specs would be very important in this ecosystem, and should become major targets for scientific debate as well as continuous updating. Instead of each paper being reviewed one by one, a field puts a lot of work into agreeing on the standards of the field instead. I can imagine multiple competing specs in a field.
There's also the issue of incentives. Currently, journal ranking and scientist rankings are very important. In order for your proposed system to work, does it fit with academics' incentives? What if the paper quality profiles got leveraged into quality indexes or epistemic indexes for scientists and journals?
Very interesting idea on field-specific specs. I’ve built out an error taxonomy for empirical articles that I’ve been using to categorize methodological critiques and generate evals. Very much in line with this idea. Re: incentives, I’ve done some thinking about this which I published recently here: https://www.paullitvak.com/p/the-future-of-academic-journals Yes, I totally agree that new systems of scientific record offer new possibilities for ways to evaluate scientific output (and scientists). There’s both promise and peril here, which I tried to address.
> At first, the amount a paper shifts the posterior on a named claim becomes a measurable and attributable way of rewarding scientists. But soon, more methods for measuring the impact of scientists proliferate. Some scientists make conceptual clarifications, while others make methodological improvements that clarify causal inference across dozens of empirical datasets at once. Others collect quality data that comprehensively adjudicates between competing theories5. There are so many different actions that are seen as furthering human and machine understanding. Scientists are rewarded for all of it.
Wow, inspiring article! This particular problem space is near and dear to my heart. You provided an excellent overview of converging technologies that could ultimately be game-changing for science if well integrated--that's the big issue by my lights. I am currently working with a team on a diagnostic tool that maps structural distortions in scientific literature in order to credibly estimate how much of the literature is wrong and promote a more productive model of research/knowledge. The challenge is as much orchestrating the right cultural and market levers as it is technological. Thanks to this article I also learned more about The Robyn Dawes Institute, a gold mine for epistemic infrastructure. We should chat!
Great question Robert! Yes, I think the structure is similar for other areas of basic science, but the kinds of audits and quality signals you’d use would be different. I wrote a broader piece on how I see this for scientific publishing as a whole here: https://www.paullitvak.com/p/the-future-of-academic-journals
Evidence synthesis
yes yes yes. if we take it to the abstract, I think te the job of making meaning out of all this information we have is one of the most important things people can do today
I love this!
I have given this some thought as well, and from my experience methodological reviews get significantly better with field-specific specs. I think these field method specs would be very important in this ecosystem, and should become major targets for scientific debate as well as continuous updating. Instead of each paper being reviewed one by one, a field puts a lot of work into agreeing on the standards of the field instead. I can imagine multiple competing specs in a field.
There's also the issue of incentives. Currently, journal ranking and scientist rankings are very important. In order for your proposed system to work, does it fit with academics' incentives? What if the paper quality profiles got leveraged into quality indexes or epistemic indexes for scientists and journals?
Very interesting idea on field-specific specs. I’ve built out an error taxonomy for empirical articles that I’ve been using to categorize methodological critiques and generate evals. Very much in line with this idea. Re: incentives, I’ve done some thinking about this which I published recently here: https://www.paullitvak.com/p/the-future-of-academic-journals Yes, I totally agree that new systems of scientific record offer new possibilities for ways to evaluate scientific output (and scientists). There’s both promise and peril here, which I tried to address.
I like it!
> At first, the amount a paper shifts the posterior on a named claim becomes a measurable and attributable way of rewarding scientists. But soon, more methods for measuring the impact of scientists proliferate. Some scientists make conceptual clarifications, while others make methodological improvements that clarify causal inference across dozens of empirical datasets at once. Others collect quality data that comprehensively adjudicates between competing theories5. There are so many different actions that are seen as furthering human and machine understanding. Scientists are rewarded for all of it.
Wow, inspiring article! This particular problem space is near and dear to my heart. You provided an excellent overview of converging technologies that could ultimately be game-changing for science if well integrated--that's the big issue by my lights. I am currently working with a team on a diagnostic tool that maps structural distortions in scientific literature in order to credibly estimate how much of the literature is wrong and promote a more productive model of research/knowledge. The challenge is as much orchestrating the right cultural and market levers as it is technological. Thanks to this article I also learned more about The Robyn Dawes Institute, a gold mine for epistemic infrastructure. We should chat!
Thanks Roger! I’d love to hear more about your tool!
Perfect, I will message you directly to set that up.
This article focuses on social science and RCT's. Any information on more basic science e.g. biomedical science?
Great question Robert! Yes, I think the structure is similar for other areas of basic science, but the kinds of audits and quality signals you’d use would be different. I wrote a broader piece on how I see this for scientific publishing as a whole here: https://www.paullitvak.com/p/the-future-of-academic-journals