Meetings that matter: the dual benefits of panel peer review

Grant peer review panels constitute a central mechanism for distributing competitive research funding, yet how such panels add value to assessments remains understudied. This exploratory analysis demonstrates that panel review can add value by increasing the extent of systematic over heuristic information processing, potentially resulting in different review outcomes than that which aggregation of individual reviews produces. On the one hand, the prospect of panel discussion can influence individual application processing, as panellists, motivated by reputational concerns, carry out more systematic individual reviews. On the other hand, panel discussion can pool and quality check panellists’ assessments, contributing further to systematic information processing. However, the extent of these benefits depends on the characteristics of the review task. The implications of these findings for review process design are examined.

• Publication year

  2024

• Venue

  Research Evaluation

• Publication date

  2024-12-27

• Fields of study

  Not labeled

• Identifiers
  DOI 10.1093/reseval/rvaf047
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Partial and particularistic: grant peer review panels evaluating interdisciplinarity

  2024cited by this paper
• Exploring the effect of overlapping institutional applications on panel decision-making

  2023cited by this paper
• Are Experts Blinded by Feasibility? Experimental Evidence from a NASA Robotics Challenge

  2022cited by this paper
• Disagreement and Agonistic Chance in Peer Review

  2021cited by this paper
• Towards theorizing peer review

  2021cited by this paper
• Face-to-face panel meetings versus remote evaluation of fellowship applications: simulation study at the Swiss National Science Foundation

  2021cited by this paper
• Conservatism Gets Funded? A Field Experiment on the Role of Negative Information in Novel Project Evaluation

  2021cited by this paper
• “Everything Is Perfect, and We Have No Problems”: Detecting and Limiting Social Desirability Bias in Qualitative Research

  2019cited by this paper
• Effective operation of competitive research funding systems

  2018cited by this paper
• What do we know about grant peer review in the health sciences?

  2017cited by this paper
• Evaluating novelty: the role of panels in the selection of R&D projects

  2017cited by this paper
• Expertise versus Bias in Evaluation: Evidence from the NIH

  2017cited by this paper
• Academic judgments under uncertainty: A study of collective anchoring effects in Swedish Research Council panel groups

  2017cited by this paper
• Looking Across and Looking Beyond the Knowledge Frontier: Intellectual Distance, Novelty, and Resource Allocation in Science

  2016cited by this paper
• What Is Naturalistic Observation

  2016cited by this paper
• Peer Review Evaluation Process of Marie Curie Actions under EU’s Seventh Framework Programme for Research

  2015cited by this paper
• A retrospective analysis of the effect of discussion in teleconference and face-to-face scientific peer-review panels

  2015cited by this paper
• The selection of talent as a group process. A literature review on the social dynamics of decision making in grant panels

  2014cited by this paper
• Seeking Qualitative Rigor in Inductive Research

  2013cited by this paper
• Bias in peer review

  2013cited by this paper
• Theory Construction in Qualitative Research

  2012cited by this paper
• A theory of heuristic and systematic information processing.

  2012cited by this paper
• Panel discussion does not improve reliability of peer review for medical research grant proposals.

  2012cited by this paper
• An Analysis of Preliminary and Post-Discussion Priority Scores for Grant Applications Peer Reviewed by the Center for Scientific Review at the NIH

  2010cited by this paper
• Panel peer review of grant applications: what do we know from research in social psychology on judgment and decision-making in groups?

  2010cited by this paper
• In Search of Synergy in Small Group Performance

  2009cited by this paper
• Motivated Information Processing in Group Judgment and Decision Making

  2008cited by this paper
• Motivated information processing and group decision-making: Effects of process accountability on information processing and decision quality☆

  2007cited by this paper
• Examining the value added by committee discussion in the review of applications for research awards

  2007cited by this paper
• Beyond blind faith: overcoming the obstacles to interdisciplinary evaluation

  2006cited by this paper
• Conclave in the Tower of Babel: how peers review interdisciplinary research proposals

  2006cited by this paper
• Assessing expert interdisciplinary work at the frontier: an empirical exploration

  2006cited by this paper
• The policy challenges of peer review: managing bias, conflict of interests and interdisciplinary assessments

  2006cited by this paper
• ORGANIZATIONS: From Input-Process-Output Models to IMOI Models

  2005cited by this paper
• Emerging Perspectives on Judgment and Decision Research: Bridging Individual, Interpersonal, and Institutional Approaches to Judgment and Decision Making: The Impact of Accountability on Cognitive Bias

  2003cited by this paper
• The Decision-Making Constraints and Processes of Grant Peer Review, and Their Effects on the Review Outcome

  2001cited by this paper
• Accounting for the effects of accountability.

  1999cited by this paper
• On the Seizing and Freezing of Negotiator Inferences: Need for Cognitive Closure Moderates the Use of Heuristics in Negotiation

  1999cited by this paper
• Basics of qualitative research: Techniques and procedures for developing grounded theory, 3rd ed.

  1998cited by this paper
• The emerging conceptualization of groups as information processors.

  1997cited by this paper
• Motivated closing of the mind: "seizing" and "freezing".

  1996cited by this paper
• Individual differences in need for cognitive closure.

  1994cited by this paper
• Heuristic processing can bias systematic processing: effects of source credibility, argument ambiguity, and task importance on attitude judgment.

  1994cited by this paper
• Motivated resistance and openness to persuasion in the presence or absence of prior information.

  1993cited by this paper
• The effects of time scarcity and time abundance on group performance quality and interaction process.

  1992cited by this paper
• Naturalistic inquiry: Beverly Hills, CA: Sage Publications, 1985, 416 pp., $25.00 (Cloth)

  1985cited by this paper
• The group polarization phenomenon.

  1976cited by this paper

• No citing papers are available for this paper.