Things I Have Learned about Meta-analysis Since 1990 - Reducing Bias in Search of “The Big Picture”

Oct 15, 2026

Uploaded by pop out

This paper examines sources of potential bias in systematic reviews and meta-analyses that can distort findings and create interpretation and application problems for practitioners and policymakers. Building on a 1990 article introducing meta-analysis for estimating population parameters and summarizing quantitative research, it analyzes cases where meta-analyses disagree. It then details substantive and methodological bias mechanisms and proposes measures to improve reliability and accuracy.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages17

Reading Time43 min

AccessFree to download

Introduction

Potential sources of bias and why results differ

Methodological and substantive pitfalls

Measures to improve reliability and accuracy

| Volume 40(3) | Summer/été 2014 | | --- | --- | Things I Have Learned about Meta-‐Analysis Since 1990: Reducing Bias in Search of “The Big Picture” Ce que j’ai appris sur la méta-‐analyse depuis 1990 : réduire les partis pris en quête d’une vue d’ensemble Robert M. Bernard, Concordia University Abstract This paper examines sources of potential bias in systematic reviews and meta-analyses which can distort their findings, leading to problems with interpretation and application by practitioners and policymakers. It follows from an article that was published in the Canadian Journal of Communication in 1990,“Integrating Research into Instructional Practice: The Use and Abuse of Meta-analysis,”which introduced meta-analysis as a means for estimating population parameters and summarizing quantitative research around instructional research questions. This paper begins by examining two cases where multiple meta-analyses disagree. It then goes on to describe substantive and methodological aspects of meta-analysis where various kinds of bias can influence the outcomes and suggests measures that can be taken to avoid them. The intention is to improve the reliability and accuracy of reviews so that practitioners can trust the results and use them more effectively. Résumé Cet article examine les sources des partis pris potentiels dans les synthèses systématiques et les méta-analyses qui peuvent déformer les conclusions, ce qui peut causer des problèmesd’interprétation et d’application par les praticiens et les responsables des politiques. Il fait suite à un article publié dans le Canadian Journal of Communication en 1990, intitulé « Integrating Research into Instructional Practice: The Use and Abuse of Meta-analysis », qui présentait laméta-analyse comme moyen d’estimer les paramètres relatifs à la population et de résumer la recherche quantitative sur les questions de recherche pédagogique. L’article commence avec l’examen de deux cas dans lesquels de nombreuses méta-analyses sont en désaccord. Il décritensuite les aspects substantifs et méthodologiques des méta-analyses dans lesquels divers types de partis pris peuvent influencer les résultats et suggère des mesures qui peuvent être adoptées pour éviter ces partis pris. L’intention est d’améliorer la fiabilité et l’exactitude des synthèsesafin que les praticiens puissent compter sur les résultats et les utiliser plus efficacement. CJLT/RCAT Vol. 40(3) Introduction In the Canadian Journal of Educational Communication (the forerunner of this journal) 1990, volume 19, number 3 (pages 171-195), Som Naidu, then a doctoral candidate in Educational Technology at Concordia University, and I co-authored an article entitled “Integrating Research into Instructional Practice: The Use and Abuse of Meta-analysis.”We hadn’t actually conducted a meta-analysis at that point, but we had read extensively about the methodology and drew examples from a variety of meta-analyses of that era, especially around the ongoing debate concerning the efficacy of mastery learning compared to regular classroom instruction. In that article, we wondered how two prominent meta-analyses (Lysakowski & Walberg, 1982; Guskey & Gates, 1986) and a “best evidence synthesis” by Robert Slavin (1987) could provide such different answers to the very same research question: “Does mastery learning produce better achievement outcomes than traditional group-based classroom instruction?” Lysakowski and Walberg reported an average effect size of around d+ = 0.94 (i.e., about one standard deviation difference between mastery learning and regular classroom) for k = 94 (i.e., 94 individual effect sizes) from 54 studies, while the Guskey and Gates meta-analysis reported a median effect size of between 0.40–0.60 for 25 studies by using more conservative methodological quality criteria than Lysakowski and Walberg. By contrast, Slavin found a median effect size of 0.0 for four studies using even more conservative inclusion c

FAQs

What problem does the paper focus on in systematic reviews and meta-analyses?

The paper focuses on potential sources of bias that can distort findings, creating difficulties for interpretation and for practical use by practitioners and policymakers.

Why do different meta-analyses sometimes reach conflicting conclusions?

The paper discusses disagreements illustrated by two cases and attributes discrepancies largely to differences in the literature selected and in inclusion/exclusion criteria.

What does the paper propose to improve the usefulness of meta-analyses?

It proposes measures to avoid bias and to improve the reliability and accuracy of reviews, helping practitioners trust results and apply them more effectively.

Oct 15, 2026

Uploaded by pop out