Reliability, fairness and predictive validity of committee peer review: Evaluation of the selection of post-graduate fellowship holders by the Boehringer Ingelheim Fonds

摘要

In science, peer review is the oldest and best established method of assessing manuscripts, applications for research fellowships and research grants. However, the fairness of peer review, its reliability and whether it achieves its aim to select the best science and scientists has often been questioned. Here we present the first comprehensive study on committee peer review for the selection of doctoral (Ph.D.) and post-doctoral research fellowships. We analysed the selection process of the Boehringer Ingelheim Fonds, a foundation for the promotion of basic research in biomedicine, with regard to its reliability, fairness and predictive validity - the three quality criteria for professional evaluations. We analysed a total of 2,697 applications, 1,954 for doctoral and 743 for post-doctoral fellowships. In 76% of the cases, the decision whether to award a scholarship or not was characterized by agreement between reviewers. Similar figures for reliability were reported for the grant selection processes of other major funding agencies. With regard to fairness, we analysed whether potential sources of bias, i.e. gender, nationality, discipline and institutional affiliation, could have influenced the decisions. For post-doctoral fellowships, no statistically significant influence of any of these variables could be observed. In applications for a doctoral fellowship, evidence of a gender, discipline and institutional bias, but not of a nationality bias, was found. We therefore present some proposals for optimizing committee peer review. The most important aspect of our study was to investigate the predictive validity of the process, i.e. whether the Foundation achieves its aim to select the best young scientists. Our bibliometric analysis showed that this is indeed the case and that the selection process is thus highly valid: research articles from Boehringer Ingelheim Fonds fellows are cited considerably more often than the »average« publication in the Institute for Scientific Information (ISI, Philadelphia, PA, USA) journal sets »Multidisciplinary«, »Molecular Biology & Cenetics«, and »Biology & Biochemistry«. These sets include journals covering the research fields in which most of the fellows publish.