On confusing prima facie validity with true validity.

摘要

Sir, Having compared the methodological quality of 28 systematic reviews on dermatological topics produced by the Cochrane Skin Group with 10 that were not, Collier et al. concluded that the Cochrane review methodology leads to higher quality reviews. For reasons to be articulated, we believe that the review by Collier et al. failed to be the critical assessment it was intended to be, primarily because it failed to criticize the fact that the reviews upon which it was based inadequately assessed trial quality. Perhaps the strongest case can be made for reviews rated a perfect 7.0 out of 7.0 by the Oxman and Guyatt Scale. For example, the review by Gibbs et al. received a rating of 7.0, and it in turn rated as high (the highest possible rating) the trial by Stender et al. on photo-dynamic therapy for hand and foot warts. The basis for this 'high' rating was 'clearly showing adequate concealment, blinding, and intention-to-treat analysis'. In fact, Stender et al. made allocation concealment quite unlikely because blocks of size two are among the easiest allocation schemes to decipher. Also, the intent-to-treat analysis was in fact not used, as seen from the exclusion of three randomized warts per group.