Systematic reviews of randomized controlled trials in literacy research : methodological challenges

摘要

Introduction: In this item a 'tertiary' review of systematic reviews in literacy learningudis presented. It explores the methodological quality of the identified systematic reviewsudand identifies the primary data that are used for the in-depth methodological work inudItem 3 on the two main threats to the validity of systematic reviews: publication biasudand design bias.udBackground: Recent governments in the UK have introduced a number of initiativesudaimed at improving the literacy levels of children. It is important, therefore, that policyudand practice are informed by the most rigorous available evidence, particularly forudquestions of effectiveness in literacy learning. It is also important that this evidence isudsubjected to rigorous critical scrutiny.udMethods: Systematic reviews undertaken in the field of literacy learning in English inudthe years between 1983 and 2003 were searched for, located and quality assessed. Theudscope of the review was limited to systematic reviews of experimental researchudevaluating literacy interventions with quantifiable literacy outcome measures in Englishudas a first (not second or additional) language and focusing on children and young peopleudin school settings up to the age of 18.udResults: A total of 14 systematic reviews containing meta-analyses and meeting all theudinclusion criteria were included in the tertiary review. The following data wereudextracted from the reviews: literacy interventions, outcomes evaluated and effect sizes.udThe quality of the reviews was examined using an adaptation of the QUORUMudstatement. Overall the quality of the meta-analyses included in this tertiary review wasudgood. When examining the effect sizes of randomized controlled trials (RCTs) andudcontrolled trials (CTs) separately there was no clear pattern as to whether the RCTsudproduced a larger or smaller effect size than the CTs.udDiscussion: Overall the quality of the meta-analyses included in this tertiary review wasudgood. The QUORUM checklist seemed to perform well for the appraisal of educationaludmeta-analyses. All the reviews clearly stated their research question, and their methodsudof searching for and selecting included studies. Most studies described their dataudextraction and used some form of quality assessment of included studies. On the otherudhand, some reviews did have notable methodological weaknesses. Six of the 14 studiesuddid not make an assessment of publication bias, which is potentially a major threat toudthe validity of any systematic review. In addition, six studies did not provide evidenceudfor reviewer agreement when synthesising the data. There is, therefore, some room forudimprovement in the methodological quality of systematic reviews in literacy learning.udConclusions: A number of reviews in this tertiary review are judged to be ofudsufficiently high quality to provide reliable evidence for the effectiveness of literacyudinterventions.ud