Measuring Student Learning in Technical Programs - A Case Study From Colombia

Oct 15, 2026

Uploaded by Cipher

Technical schools play an integral role in higher education, yet student learning in these settings remains understudied. This case study examines whether assessments can be jointly administered to university and technical school students without biasing inferences about relative performance in quantitative reasoning and critical reading. Using item response theory models, the study identifies differential test functioning that favors university students and quantifies how it distorts effect sizes and institution rank orderings based on SABER PRO data from Colombia.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages11

Reading Time28 min

AccessFree to download

Background and Motivation

Research Questions

Data and Assessment Context

Methodology: Item Response Theory

Differential Test Functioning Findings

Implications for Effect Sizes and Rankings

AERA Open January-March 2017, Vol. 3, No. 1, pp. 1–11 DOI: 10. 1177/2332858417692997 © The Author(s) 2017. [http://ero.sagepub.com](http://ero.sagepub.com) Measuring Student Learning in Technical Programs: A Case Study From Colombia Benjamin W. Domingue David Lang Stanford University Martha Cuevas Instituto Colombiano para la Evaluación de la Educación Melisa Castellanos Concordia University Carolina Lopera Julián P. Mariño Adriana Molina Universidad de los Andes Richard J. Shavelson Stanford University Technical schools are an integral part of the education system, and yet, little is known about student learning at such institutions. We consider whether assessments of student learning can be jointly administered to both university and technical school students. We examine whether differential test functioning may bias inferences regarding the relative performance of students in quantitative reasoning and critical reading. We apply item response theory models that allow for differences in response behavior as a function of school context. Items show small yet consistent differential functioning in favor of university students, especially for the quantitative reasoning test. These differences are shown to affect inferences regarding effect size differences between the university and technical students (effect sizes can fall by 44% in quantitative reasoning and 24% in critical reading). Differential test functioning influences the rank orderings of institutions by up to roughly 5 percentile points on average. Keywords: technical schools, community colleges, Colombia, higher education, differential test functioning Globally, higher education is a crucial component of many students’paths to professional success. However, in only a few countries is the learning that occurs in higher educational institutions systematically assessed (Zlatkin-Troitschanskaia, Shavelson, & Kuhn, 2015) . Moreover, existing assessments tend to focus on universities.“Technical schools”—which we define broadly to include community colleges, junior colleges, and technical schools—are an important part of the higher education system that offer both important vocational training and certification and, for many students, preparation for enrollment in traditional institutions of higher learning. There is limited research available about these institutions, especially about their effect on student learning. Expanding this limited research base is important because technical schools serve large numbers of students, many of whom are underrepresented in traditional higher educational institutions. For example, they are likely to be of lower socioeconomic status as compared to students of traditional universities. An additional concern is that for-profit institutions have become major providers in this space and are frequently of dubious quality (e.g., Deming, Goldin, & Katz, 2011) . An understanding of the impact these institutions have on learning is clearly needed. The development of specialized instruments to measure learning outcomes in technical schools would be costly and time-consuming. A natural alternative, especially given the complementary nature of their missions, is to design instruments to measure the learning of both university and technical school students. Although this solution is promising from a budgetary perspective, caution is required because university and technical students may encounter dramatically different pedagogical environments (e.g., study of the thermodynamic principles that underlie engines for university students versus Creative Commons CC-BY: This article is distributed under the terms of the Creative Commons Attribution 3.0 License ([http://www.creativecommons.org/licenses/by/3.0/](http://www.creativecommons.org/licenses/by/3.0/)) which permits any use, reproduction and distribution of the work without further permission provided the original work is attributed as specified on the

FAQs

Why study student learning in technical programs?

Technical schools are central to higher education, but research on how they affect student learning is limited, especially compared with universities.

What assessment data does the study use in Colombia?

The study uses SABER PRO, Colombia’s universal assessment of quantitative reasoning and critical reading competencies at the end of postsecondary schooling.

How does differential test functioning affect the study’s conclusions?

Differential test functioning favors university students and reduces inferred effect size differences, altering both mean performance interpretations and institution rank orderings.

Oct 15, 2026

Uploaded by Cipher