The CORD Standardized Letter of Evaluation: Have We Achieved Perfection or Just a Better Understanding of Our Limitations?

摘要

In the early 1990s, an emergency medicine (EM) program director remediated a resident for over a year, to no avail. The resident's contract was not renewed, and a recommendation made that the resident consider another specialty. When this decision was discussed with the department chair and clerkship director, who had written a very positive and “flowery” narrative letter of recommendation (NLOR), the chair said, “We knew this resident would struggle.” The “fluffed up” letter was a disservice to colleagues and to the resident, who spent a difficult year in over her head. At the time, the general discussion among EM program directors indicated that accurate information transfer was a common limitation of the NLOR. Often, NLORs included no objective data (not even the EM clerkship grade) and provided no global comparison to other students. It was perceived that, often, one could not get to a “bottom line” view of the candidate despite a lengthy letter that was time consuming to prepare. In an attempt to address the problems with NLORs, a council of EM residency directors (CORD) subcommittee developed a standardized letter of recommendation (SLOR) in 1995, which was initiated in 1997.1 The SLOR offered more objective data than the NLOR, including an evaluation of EM clerkship performance and a prediction by the writer of how their program might rank the student. The original SLOR included the following 4 sections: (A) background information (clerkship performance); (B) qualifications for EM (personal characteristics relating to the choice of EM); (C) global assessment (comparisons to other students); and (D) written comments. As discussed in a paper by Girzadas et al,2 it quickly became clear that the SLOR was easier to prepare and read with the new format. Whether it made information transfer more reliable was a separate question, and that led to research of potential SLOR limitations. That research, similar to what Girzadas et al2 had found, suggested some problems.3,4 Potential biases resulting in grade inflation of the SLOR were uncovered. Areas of concern included gender bias, inexperience by letter writers, and the duration of time the letter writer knew the applicant.3,4 Another paper by a CORD task force demonstrated evidence of SLOR “grade inflation,” as 40% of reviewed SLORs rated their applicants in the “top 10%,” and over 95% of these SLORs rated applicants in the top third.5 Finally, when rank lists were compared with the global assessment question regarding estimated rank list position, overestimation on the SLOR occurred 66% of the time.6 The SLOR is central to 2 papers in this issue of the Journal of Graduate Medical Education. The study by Diab et al7 shows that the SLOR, with its measurable categories, allows research into the application process. Diab et al7 demonstrated a significant increase in the global assessment ranking of “outstanding” in letters where applicants did not waive their Family Educational Rights and Privacy Act (FERPA) rights, suggesting that if a faculty member is aware that an applicant may read their SLOR, the grade may be inflated.7 Thankfully, 93% of applicants waived their FERPA rights. The study is limited in that we do not know whether applicants who did not waive their rights were representative of the whole population of applicants, but it does suggest one should consider the possibility of bias if no waiver is present. The paper by Hegarty et al8 describes the work of a CORD SLOR task force that was convened in 2011 to review the SLOR and determine whether improvements could be recommended. Although only 37% of the group surveyed had read the CORD guidelines in the previous year, these guidelines were very general and did not include specific recommendations for each question or even for each of the 4 sections. The consequence was great variability in how the question regarding “One Key Comment from ED Faculty Evaluations” was addressed in section A. The manner in which answers we