Abstract
The Canadian Institutes of Health Research (CIHR) commenced a Quality Assurance Program in 2019 to monitor the quality of peer review in its Project Grant Competition Peer Review Committees. Our primary aim was to describe the performance of CIHR grant peer reviewers, based on the assessments made by CIHR peer review leaders during the first 3 years of the Research Quality Assurance Program. All Peer Review Committee Chairs and (or) Scientific Officers who led peer review for CIHR in 2019, 2020, and 2021 completed Reviewer Quality Feedback forms immediately following Peer Review Committee meetings. The form assessed Performance, Future potential, Review quality, Participation, and Responsiveness. We summarised and descriptively synthesised data from assessments conducted after each of the four grant competitions. The performance of peer reviewers on 4438 occasions was rated by Chairs and Scientific Officers. Approximately one in three peer reviewers submitted outstanding reviews or discussed additional applications and one in 10 demonstrated potential as a future Peer Review Committee leader. At most, one in 20 peer reviewers was considered to have not performed adequately with respect to review quality, participation, or responsiveness. There is a need for more research on the processes involved in allocating research grant funding.
Introduction
Research funding is a major determinant of what scientists study, yet there has been relatively little research into the various methods that major funders use to award competitive grants. Evidence should inform how funders design, deliver, monitor, and evaluate grant peer review (Shepherd et al. 2018), yet very few national health research funders have evaluated the quality of their grant peer review.
The European Science Foundation Survey Analysis Report on Peer Review Practices mentions 25 funders that evaluate the quality and useability of grant peer review reports as standard practice (European Science Foundation 2011a). A small number of funders appear to use formal quality assurance processes covering quality indicators or expectations of quality peer review/peer reviewers. Some funders may commission external auditors to conduct internal quality assurance of their processes. Quality assurance is nominated as a central pillar of best practice in grant peer review (European Science Foundation 2011b). Several tools have been developed to assess the quality of journal peer review, although none have sufficient validity to warrant widespread use at this time (Superchi et al. 2019). We are unaware of published tools for assessing the quality of grant peer review.
Systematic evaluation keeps funders accountable to the communities they serve and demonstrates commitment to quality improvement (European Science Foundation 2012). In 2019, after a 2-year pilot phase, the Canadian Institutes of Health Research (CIHR) commenced a continuous quality assurance program (Research Quality Assurance Process) to routinely monitor the quality of peer reviewers’ participation in and contributions to Project Grant Competition Peer Review Committees (Canadian Institutes of Health Research 2022e). Peer review leaders (committee Chairs and Scientific Officers) evaluated the performance of each Peer Review Committee member contributing to delivering CIHR peer review, according to criteria developed and refined by CIHR over the pilot phase.
Regular audit of performance would help to establish whether funders meet their aspirations in delivering peer review (European Science Foundation 2011b). Yet, few funders institute formal monitoring of the quality of grant peer review (e.g., quality assurance) (European Science Foundation 2011a; Guthrie et al. 2018; Recio-Saucedo et al. 2022).
The primary aim of this study was to describe the performance of grant peer reviewers in the Canadian health research funding system, based on the assessments made by CIHR peer review leaders, during the first 3 years of the Research Quality Assurance Process. The secondary aims were to describe the performance of grant peer reviewers by sex, career stage, university affiliation, and geographic location.
