An Examination of the Performance of Variance Estimators in International Large-Scale Assessments - Final Report

Oct 15, 2026

Final report analyzing the performance of variance estimators in international large-scale assessments (ILSA). Presents a theoretical framework covering standard errors, variance estimators, variance strata, and replication methods including Jackknife repeated replication (JK2), balanced repeated replication (BRR), and bootstrapping. Describes simulation studies with sampling design, replicate weights, and parameter estimation, then evaluates relative bias and stability. Reports results using means, subgroup comparisons, percentiles, benchmark percentages, and correlations across multiple scenarios, concluding with discussion and recommendations.

LanguageEnglish

CategoryExam

FormatPDF

Pages93

Reading Time3 h 54 min

AccessFree to download

Abstract

Introduction

Theoretical Framework

Standard Errors in ILSAs

Variance Estimators

Variance Strata

Replication Methods

Differences in Building Variance Strata in ILSAs

The Current Study: Research Questions

Methods

Simulated Population

Sampling Design and Weights

Variance Estimators and Replicate Weights

Number of Replicate Weights

Estimated Parameters of Interest

Relative Bias and Stability of Variance Estimators

Results

Means (Overall)

Sub-Group Means (Gender = Girls)

Differences in Sub-Group Means (Girls-Boys)

Percentiles (25th, 50th, 75th)

Benchmark Percentages

Correlations

Summary across all Results

Discussion and Recommendations

An Examination of the Performance of Variance Estimators in International Large-Scale Assessments Final Report 2025.07.07 Umut Atasever, Sabine Meinck, Diego Cortes 1 1 All authors are affiliated with IEA, Hamburg, Germany. Table of Contents 1 Abstract .................................................................................................................................................. 4 2 Introduction ............................................................................................................................................ 6 3 Theoretical Framework .......................................................................................................................... 9 3.1 Standard Errors in ILSAs............................................................................................................... 9 3.2 Variance Estimators ..................................................................................................................... 10 3.3 Variance Strata ............................................................................................................................ 11 3.4 Replication Methods .................................................................................................................... 13 3.4.1 Jackknife Repeated Replication (JK2) ..................................................................................... 13 3.4.2 Balanced Repeated Replication (BRR) .................................................................................... 15 3.4.3 Bootstrapping .......................................................................................................................... 19 3.5 Differences in Building Variance Strata in ILSAs.......................................................................20 3.5.1 Non-Response ......................................................................................................................... 20 3.5.2 Odd number of PSUs in the explicit stratum ........................................................................... 21 3.6 Current State of Research ............................................................................................................22 4 The Current Study: Research Questions ..............................................................................................24 5 Methods................................................................................................................................................27 5.1 Simulated Population ...................................................................................................................28 5.2 Sampling Design and Weights .....................................................................................................31 5.3 Variance Estimators and Replicate Weights ................................................................................35 5.3.1 Jackknife Repeated Replication (JK2) ..................................................................................... 35 5.3.2 Balanced Repeated Replication (BRR) .................................................................................... 36 5.3.3 Bootstrapping .......................................................................................................................... 37 5.4 Number of Replicate Weights ......................................................................................................39 5.5 Estimated Parameters of Interest .................................................................................................41 5.6 Relative Bias and Stability of Variance Estimators .....................................................................43 6 Results..................................................................................................................................................44 6.1 Means (Overall) ...........

FAQs

What variance estimation approaches are evaluated in the report?

The report evaluates Jackknife repeated replication (JK2), balanced repeated replication (BRR), and bootstrapping, focusing on how they estimate variance in ILSAs.

How does the study assess performance of variance estimators?

Performance is assessed through simulation, including sampling design and weights, estimation of parameters of interest, and evaluation of relative bias and stability of variance estimators.

What types of results are presented in the findings section?

Results include overall means, subgroup means and differences, percentiles, benchmark percentages, correlations, and scenario-based summaries such as regular sample, odd number of schools, and non-response.

Oct 15, 2026

Uploaded by Maya Linwood