Meta-analytic Decisions and Reliability: A Serendipitous Case of Three Independent Telecommuting Meta-analyses

摘要

Despite the potential for researcher decisions to negatively impact the reliability of meta-analysis, very few methodological studies have examined this possibility. The present study compared three independent and concurrent telecommuting meta-analyses in order to determine how researcher decisions affected the process and findings of these studies. Methodology: A case study methodology was used, in which three recent telecommuting meta-analyses were re-examined and compared using the process model developed by Wanous et al. (J Appl Psychol 74:259-264, 1989). Findings: Results demonstrated important ways in which researcher decisions converged and diverged at stages of the meta-analytic process. The influence of researcher divergence on meta-analytic findings was neither evident in all cases, nor straightforward. Most notably, the overall effects of telecommuting across a range of employee outcomes were generally consistent across the meta-analyses, despite substantial differences in meta-analytic samples. Implications: Results suggest that the effect of researcher decisions on meta-analytic findings may be largely indirect, such as when early decisions guide the specific moderation tests that can be undertaken at later stages. However, directly comparable "main effect" findings appeared to be more robust to divergence in researcher decisions. These results provide tentative positive evidence regarding the reliability of meta-analytic methods and suggest targeted areas for future methodological studies. Originality: This study presents unique insight into a methodological issue that has not received adequate research attention, yet has potential implications for the reliability and validity of meta-analysis as a method.