Examining the Predictive Validity of NIH Peer Review Scores

The predictive validity of peer review at the National Institutes of Health (NIH) has not yet been demonstrated empirically. It might be assumed that the most efficient and expedient test of the predictive validity of NIH peer review would be an examination of the correlation between percentile scores from peer review and bibliometric indices of the publications produced from funded projects. The present study used a large dataset to examine the rationale for such a study, to determine if it would satisfy the requirements for a test of predictive validity. The results show significant restriction of range in the applications selected for funding. Furthermore, those few applications that are funded with slightly worse peer review scores are not selected at random or representative of other applications in the same range. The funding institutes also negotiate with applicants to address issues identified during peer review. Therefore, the peer review scores assigned to the submitted applications, especially for those few funded applications with slightly worse peer review scores, do not reflect the changed and improved projects that are eventually funded. In addition, citation metrics by themselves are not valid or appropriate measures of scientific impact. The use of bibliometric indices on their own to measure scientific impact would likely increase the inefficiencies and problems with replicability already largely attributed to the current over-emphasis on bibliometric indices. Therefore, retrospective analyses of the correlation between percentile scores from peer review and bibliometric indices of the publications resulting from funded grant applications are not valid tests of the predictive validity of peer review at the NIH.

• Publication year

  2015

• Venue

  PLoS ONE

• Publication date

  2015-06-03

• Fields of study

  Medicine, Psychology

• Identifiers
  DOI 10.1371/journal.pone.0126938PMID 26039440PMCID 4454673
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar, PubMed

• No claims are published for this paper.

• No concepts are published for this paper.

• Exploratory data analysis

  2018cited by this paper
• Discontinuation and non-publication of surgical randomised controlled trials: observational study

  2014cited by this paper
• Percentile Ranking and Citation Impact of a Large Cohort of National Heart, Lung, and Blood Institute–Funded Cardiovascular R01 Grants

  2014cited by this paper
• NIH inventions translate into drugs and biologics with high public health impact

  2014cited by this paper
• Biomedical research: increasing value, reducing waste.

  2014cited by this paper
• The Validation of Peer Review through Research Impact Measures and the Implications for Funding Strategies

  2014cited by this paper
• Prior Publication Productivity, Grant Percentile Ranking, and Topic-Normalized Citation Impact of NHLBI Cardiovascular R01 Grants

  2014cited by this paper
• Policy: NIH plans to enhance reproducibility

  2014cited by this paper
• Increasing value and reducing waste in research design, conduct, and analysis.

  2014cited by this paper
• NIH plans to enhance reproducibility

  2014cited by this paper
• United States life tables, 2009.

  2014cited by this paper
• Research: increasing value, reducing waste 1 How to increase value and reduce waste when research priorities are set

  2014cited by this paper
• The predictive power of NSF reviewers and panels

  2013cited by this paper
• Power failure: why small sample size undermines the reliability of neuroscience

  2013cited by this paper
• A Perverted View of “Impact”

  2013cited by this paper
• Publishing pays: Economists’ salaries reflect productivity

  2013cited by this paper
• Citation bias favoring statistically significant studies was present in medical research.

  2013cited by this paper
• United States Life Tables

  2013cited by this paper
• Research grants: Conform and be funded

  2012cited by this paper
• Intended and unintended consequences of a publish-or-perish culture: A worldwide survey

  2012cited by this paper
• Drug development: Raise standards for preclinical cancer research

  2012cited by this paper
• Replication and reproducibility in spinal cord injury research.

  2012cited by this paper
• The Rules of the Game Called Psychological Science

  2012cited by this paper
• False-Positive Psychology

  2011cited by this paper
• Iron, zinc and copper in the Alzheimer's disease brain: a quantitative meta-analysis. Some insight on the influence of citation bias on scientific opinion.

  2011cited by this paper
• Interventions for latent autoimmune diabetes (LADA) in adults.

  2011cited by this paper
• A Systematic Examination of the Citation of Prior Research in Reports of Randomized, Controlled Trials

  2011cited by this paper
• Is there evidence of publication biases in JDM research?

  2011cited by this paper
• The role of public-sector research in the discovery of drugs and vaccines.

  2011cited by this paper
• The outflow of academic papers from China: why is it happening and can it be stemmed?

  2011cited by this paper
• Pitfalls of Using Citation Indices for Making Academic Accounting Promotion, Tenure, Teaching Load, and Merit Pay Decisions

  2011cited by this paper
• Negative results are disappearing from most disciplines and countries

  2011cited by this paper
• The outflow of academic papers from China: why is it happening and can it be stemmed?

  2011cited by this paper
• Changing Incentives to Publish

  2011cited by this paper
• Believe it or not: how much can we rely on published data on potential drug targets?

  2011cited by this paper
• Metrics: Do metrics matter?

  2010cited by this paper
• Do Pressures to Publish Increase Scientists' Bias? An Empirical Support from US States Data

  2010cited by this paper
• Sources of funding for Nobel Prize‐winning work: public or private?

  2010cited by this paper
• Bias in genetic association studies and impact factor

  2009cited by this paper
• Avoidable Waste in the Production and Reporting of Research Evidence

  2009influential reference
• How citation distortions create unfounded authority: analysis of a citation network

  2009cited by this paper
• Learning from the Experience of Others: The Evolution of Faculty Tenure and Promotion Rules in Comprehensive Institutions

  2009cited by this paper
• A field synopsis on low-penetrance variants in DNA repair genes and cancer susceptibility.

  2009cited by this paper
• An assessment of promotion and tenure requirements at dental schools.

  2009cited by this paper
• Citation Bias in Reported Smoking Prevalence in People with Schizophrenia

  2009cited by this paper
• When Knowledge Wins: Transcending the Sense and Nonsense of Academic Rankings

  2009cited by this paper
• Factors and indices are one thing, deciding who is scholarly, why they are scholarly, and the relative value of their scholarship is something else entirely

  2008cited by this paper
• Statistical analysis of the National Institutes of Health peer review system

  2008cited by this paper
• Why Current Publication Practices May Distort Science

  2008cited by this paper
• Why Most Discovered True Associations Are Inflated

  2008cited by this paper
• Ethyl pyruvate decreased early nuclear factor-&kgr;B levels but worsened survival in lipopolysaccharide-challenged mice*

  2008cited by this paper
• Factors and indices are one thing, deciding who is scholarly, why they are scholarly, and the relative value of their scholarship is something else entirely

  2008cited by this paper
• Lost in publication: how measurement harms science

  2008cited by this paper
• Publish or Perish? The Increasing Importance of Publications for Prospective Economics Professors in Austria, Germany and Switzerland

  2008cited by this paper
• Quality journals and gamesmanship in management studies

  2008cited by this paper
• Core Values and Evaluation Processes Associated with Academic Tenure

  2007cited by this paper
• Clinical attrition due to biased preclinical assessments of potential efficacy.

  2007cited by this paper
• The Perverse Effects of Competition on Scientists’ Work and Relationships

  2007cited by this paper
• POINT: Statistical analysis in NIH peer review— identifying innovation

  2007cited by this paper
• Aardvark et al.: quality journals and gamesmanship in management studies

  2007cited by this paper
• Peer review for improving the quality of grant applications.

  2007cited by this paper
• Lost in translation: Treatment trials in the SOD1 mouse and in human ALS

  2007cited by this paper
• Hypothermia in animal models of acute ischaemic stroke: a systematic review and meta-analysis.

  2007cited by this paper
• Bench to Bedside: The Quest for Quality in Experimental Stroke Research

  2006cited by this paper
• Change in chronic disability from 1982 to 2004/2005 as measured by long-term changes in function and health in the U.S. elderly population

  2006cited by this paper
• Microarrays and molecular research: noise discovery?

  2005cited by this paper
• Contradicted and initially stronger effects in highly cited clinical research.

  2005cited by this paper
• Is citation analysis a legitimate evaluation tool?

  2005cited by this paper
• Beyond the Rhetoric: Trends in the Relative Value of Teaching and Research in Faculty Salaries

  2005cited by this paper
• Why Most Published Research Findings Are False

  2005cited by this paper
• Scientists behaving badly

  2005cited by this paper
• Establishment of genetic associations for complex diseases is independent of early study findings

  2004cited by this paper
• The politics of publication

  2003cited by this paper
• Meta-analysis of genetic association studies supports a contribution of common variants to susceptibility to common disease

  2003cited by this paper
• Assessing Publication for Tenure

  2003cited by this paper
• Citation bias of hepato-biliary randomized clinical trials.

  2002cited by this paper
• HARKing: Hypothesizing After the Results are Known

  2002cited by this paper
• Reputational Capital and Academic Pay

  2001cited by this paper
• Replication validity of genetic association studies

  2001cited by this paper
• Medical R&D at the turn of the millennium.

  2001cited by this paper
• Association study designs for complex diseases

  2001cited by this paper
• An Analysis of the Critical Role of Public Science in Innovation: The Case of Biotechnology

  2000cited by this paper
• HARKing: Hypothesizing After the Results are Known

  1998cited by this paper
• A Half Century of Peer Review, 1946-1996 (review)

  1998cited by this paper
• Do Academic Salaries Decline with Seniority?

  1998cited by this paper
• A half‐century of peer review, 1946–1996

  1998cited by this paper
• Clinical trials with multiple outcomes: a statistical perspective on their design, analysis, and interpretation.

  1997cited by this paper
• The increasing linkage between U.S. technology and public science

  1997cited by this paper
• Scholarly accomplishments in promotion and tenure decisions of accounting faculty

  1994cited by this paper
• The miracle of DICE therapy for acute stroke: fact or fictional product of subgroup analysis?

  1994cited by this paper
• Publication Bias: The Problem That Won't Go Away

  1993cited by this paper
• Faculty review criteria in tourism and hospitality

  1990cited by this paper
• Changing Order: Replication and Induction in Scientific Practice by H. M. Collins (review)

  1988cited by this paper
• Publication bias and clinical trials.

  1987cited by this paper
• Reference bias in reports of drug trials.

  1987cited by this paper
• What is a Citation Worth

  1986cited by this paper
• Private acts and public objects: An investigation of citer motivations

  1985cited by this paper
• Understanding Robust and Exploratory Data Analysis

  1984cited by this paper
• Scholarships, Citations and Salaries: Economic Rewards in Economics

  1982cited by this paper
• Exploratory Data Analysis.

  1978cited by this paper

• Funding Research Randomly

  2025cites this paper
• Targeted, actionable and fair: Reviewer reports as feedback and its effect on ECR career choices

  2023cites this paper
• Study on the proposal evaluation system for the EU R&I framework programme Final report

  2022cites this paper
• Refinement: Measuring informativeness of ratings in the absence of a gold standard.

  2022cites this paper
• Assessing quality of selection procedures: Lower bound of false positive rate as a function of inter-rater reliability.

  2022cites this paper
• Is research funding always beneficial? A cross-disciplinary analysis of U.K. research 2014–20

  2022cites this paper
• When zero may not be zero: A cautionary note on the use of inter‐rater reliability in evaluating grant peer review

  2021influential citation
• A retrospective analysis of the peer review of more than 75,000 Marie Curie proposals between 2007 and 2018

  2021cites this paper
• On the very idea of pursuitworthiness.

  2021cites this paper
• Reading Peer Review

  2020cites this paper
• fetishisation of excellence

  2020cites this paper
• Measuring bias, burden and conservatism in research funding processes

  2019cites this paper
• Peer review of health research funding proposals: A systematic map and systematic review of innovations for effectiveness and efficiency

  2018cites this paper
• Addressing Reproducibility: Peer Review, Impact Factors, Checklists, Guidelines, and Reproducibility Initiatives

  2018cites this paper
• Recommendations for Reviewers of Biomedical Imaging Grant Applications

  2018cites this paper
• Do funding applications where peer reviewers disagree have higher citations? A cross-sectional study.

  2018cites this paper
• Current Referee Status : Version 1 24 September 2018

  2018cites this paper
• Unique Review Criteria and Patient and Stakeholder Reviewers: Analysis of PCORI's Approach to Research Funding.

  2018cites this paper
• External Tests of Peer Review Validity Via Impact Measures

  2018influential citation
• An analysis of discontinued journals by Scopus

  2018cites this paper
• Toward predicting research proposal success

  2018cites this paper
• Peer Review Practices for Evaluating Biomedical Research Grants: A Scientific Statement From the American Heart Association.

  2017cites this paper
• Erratum: “Excellence R Us”: university research and the fetishisation of excellence

  2017cites this paper
• “Excellence R Us”: university research and the fetishisation of excellence

  2017cites this paper
• NIH peer review percentile scores are poorly predictive of grant productivity

  2016cites this paper
• Correction: Examining the Predictive Validity of NIH Peer Review Scores

  2015cites this paper
• Reviewing Peer Review at the NIH.

  2015cites this paper