People rag on peer review a lot (including, occasionally, New Things Under the Sun). Yet it remains one of the most common ways to allocate scientific resources, whether those be R&D dollars or slots in journals. Is this all a mistake? Or does peer review help in its purported goal to identify the science most likely to have an impact and hence, perhaps most deserving of some of those limited scientific resources?
A simple way to check is to compare peer review scores to other metrics of subsequent scientific impact; does peer review predict eventual impact?
A number of studies find it does.
This podcast is an audio read through of the (initial version of the) article What does peer review know?, originally published on New Things Under the Sun.
Articles mentioned
Li, Danielle, and Leila Agha. 2015. Big names or big ideas: Do peer-review panels select the best science proposals? Science 348(6233): 434-438. https://doi.org/10.1126/science.aaa0185
Park, Hyunwoo, Jeongsik (Jay) Lee, and Byung-Cheol Kim. 2015. Project selection in NIH: A natural experiment from ARRA. Research Policy 44(6): 1145-1159. https://doi.org/10.1016/j.respol.2015.03.004.
Card, David, and Stefano DellaVigna. 2020. What do Editors Maximize? Evidence from Four Economics Journals. The Review of Economics and Statistics 102(1): 195-217. https://doi.org/10.1162/rest_a_00839
Siler, Kyle, Kirby Lee, and Lisa Bero. 2014. Measuring the effectiveness of scientific gatekeeping. PNAS 112(2): 360-365. https://doi.org/10.1073/pnas.1418218112
Teplitskiy, Misha, and Von Bakanic. 2016. Do Peer Reviews Predict Impact? Evidence from the American Sociological Review, 1978 to 1982. Socius, 2. https://doi.org/10.1177/2378023116640278
People rag on peer review a lot (including, occasionally, New Things Under the Sun). Yet it remains one of the most common ways to allocate scientific resources, whether those be R&D dollars or slots in journals. Is this all a mistake? Or does peer review help in its purported goal to identify the science most likely to have an impact and hence, perhaps most deserving of some of those limited scientific resources?
A simple way to check is to compare peer review scores to other metrics of subsequent scientific impact; does peer review predict eventual impact?
A number of studies find it does.
This podcast is an audio read through of the (initial version of the) article What does peer review know?, originally published on New Things Under the Sun.
Articles mentioned
Li, Danielle, and Leila Agha. 2015. Big names or big ideas: Do peer-review panels select the best science proposals? Science 348(6233): 434-438. https://doi.org/10.1126/science.aaa0185
Park, Hyunwoo, Jeongsik (Jay) Lee, and Byung-Cheol Kim. 2015. Project selection in NIH: A natural experiment from ARRA. Research Policy 44(6): 1145-1159. https://doi.org/10.1016/j.respol.2015.03.004.
Card, David, and Stefano DellaVigna. 2020. What do Editors Maximize? Evidence from Four Economics Journals. The Review of Economics and Statistics 102(1): 195-217. https://doi.org/10.1162/rest_a_00839
Siler, Kyle, Kirby Lee, and Lisa Bero. 2014. Measuring the effectiveness of scientific gatekeeping. PNAS 112(2): 360-365. https://doi.org/10.1073/pnas.1418218112
Teplitskiy, Misha, and Von Bakanic. 2016. Do Peer Reviews Predict Impact? Evidence from the American Sociological Review, 1978 to 1982. Socius, 2. https://doi.org/10.1177/2378023116640278