PITFALLS AND POTENTIAL OF SECONDARY DATA ANALYSIS - IN EDUCATION ASSESSMENTS OF THE COUNCIL OF MINISTERS OF EDUCATION, CANADA

Oct 15, 2026

The article examines the pitfalls and potential of conducting secondary data analysis using test and survey data from the Council of Ministers of Education, Canada and provincial testing agencies. It outlines why deeper, more comprehensive analyses can extend policy-relevant understanding beyond initial primary summaries, enabling more questions and higher cost effectiveness. It also details seven key pitfalls, including sampling limits, missing variables, weak linkages across levels, nonresponse missing data, inaccurate responses, and incomplete documentation. The authors argue for stronger collaboration with qualified secondary analysts to support valid, useful findings.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages14

Reading Time35 min

AccessFree to download

Introduction

Potential of Secondary Data Analysis

Primary vs. Secondary Analysis

Evidence from Canadian, National, and International Testing

Mixed Experiences in Canada

Data and Documentation Problems

Authors’ Position

Large-Scale Secondary Analyses (2001 Math, 2003 Writing)

Empirical Models and Policy-Relevant Framework

PITFALLS AND POTENTIAL OF SECONDARY DATA ANALYSIS OF THE COUNCIL OF MINISTERS OF EDUCATION, CANADA, NATIONAL ASSESSMENT W. Todd Rogers, John O. Anderson, Don A. Klinger, & Teresa Dawber LargeȬscale scale testing agencies, which are generally located within provincial ministries of education and the Council of Ministers of Education, Canada, have called for secondary data analysis of test and survey data that link contextual and educational factors to student performance. First, the potential of secondary analyses is outlined, followed by discussion of seven pitfalls that need to be addressed to realize this potential in Canada. Greater collaboration between testing agencies and teams of qualified secondary analysis researchers needs to take place to ensure that policy issues relevant to education are addressed in meaningful ways. Key words: largeȬscale testing, policy analysis, missing data Les organismes responsables des épreuves communes, qui relèvent généralement des ministères provinciaux de l’Éducation et du Conseil des ministres de lȇÉducation (Canada), ont demandé une analyse de deuxième ordre des données d’épreuves etd’enquêtes reliant des facteurs contextuels et pédagogiques au rendement des élèves. Le potentiel des analyses de deuxième ordre est d’abord présenté ; suit une discussion des sept écueils à éviter pour en tirer profit au Canada. Une meilleure collaboration entre les organismes responsables des tests et les équipes de chercheurs spécialisés dans les analyses de deuxième ordre doit être mise en place afin d’assurer que les questions de politiques en matière d’éducation sont abordées de manière pertinente. Mots clés : épreuves communs, analyse de politiques, données manquantes _________________ CANADIAN JOURNAL OF EDUCATION 29, 3 (2006): 757Ȭ770 LargeȬscale testing agencies, which in Canada are generally located within provincial ministries of education and the Council of Ministers of Education, have called for secondary data analysis of test and survey data that link contextual and educational factors to student performance. Testing agencies complete the first or primary analysis of the data collected. These analyses generally consist of summary statistics aggregated at student, school, district, and/or provincial levels, and are completed either by personnel within the agencies or by external contractors. However, agency officials believe that the data can be analyzed in different ways and that deeper, more comprehensive analyses that relate variables at the student, class, and school levels to student achievement measures can be conducted. Consequently, they make data available to independent researchers to perform secondary data analyses designed to better answer the original questions or to answer new questions. Such secondary analyses increase the cost effectiveness of testing programs because data then serve a broader spectrum of research questions and researchers. The primary research questions are addressed and also a series of secondary, more complex and deeper questions. Answers to these questions are potentially of greater interest to administrators and officers of testing agencies and the clients they serve. Typically, secondary data analysts, who are located at universities, are independently funded, thereby adding an aura of autonomy and credibility to the secondary results and findings. It is possible to see secondary data analyses of provincial testing data (e.g., British Columbia, Alberta, and Ontario), national testing data (the literacy, mathematics, writing, and science assessments conducted by the Council of Ministers of Education), and international testing data (e.g., Third International Mathematics and Science Studies [TIMMS] conducted by the International Education Association and the Programme for International Student Assessment [PISA] conducted by the Organization of Economic and Cooperative Development) . Statistics Canada encourages secondary data analyses t

FAQs

What does secondary data analysis add beyond the primary analysis by testing agencies?

Secondary analyses enable deeper modeling that links variables across student, class, and school levels to achievement measures, allowing both the original and new research questions to be answered more broadly and cost-effectively.

What are the main pitfalls to address in Canada?

The article highlights seven pitfalls, including restrictive sampling designs, failure to collect needed variables, weak linkages between contextual and achievement data, missing data from nonresponse, inaccurate responses, and incomplete or incorrect documentation.

How do the authors’ secondary analyses connect to policy and practice?

They develop and validate empirical models relating educational resources and conditions, and home and school traits, to significant learning outcomes within a policy-relevant framework to better support understanding of school performance.

Oct 15, 2026

Uploaded by Alexandar