Measuring Program Quality - Evidence of the Scientific Validity of the Assessment of Program Practices Tool

Oct 15, 2026

Uploaded by Stanley

Out-of-school time (OST) youth programs are difficult to assess because they are dynamic and occur across varied settings with frequent staff–youth interactions. Measuring program quality is presented as essential for guiding directors and policymakers on where to improve and how to support improvement. The article reports research validating the Assessment of Program Practices Tool (APT), including two validation phases that examine scientific rigor, support tool improvements, and inform training to strengthen rating reliability for high-stakes users.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages9

Reading Time23 min

AccessFree to download

Measuring Program Quality

Why Program Quality Matters in OST Programs

Assessment of Program Practices Tool (APT)

APT Domains and Quality Areas

Rating Scale, Anchors, and Conditional Items

APT Validation Study Phase 1: Scientific Rigor

Methods and Participating Programs

NEW FROM NIOST Measuring Program Quality: Evidence of the Scientiﬁc Validity of the Assessment of Program Practices Tool Allison Tracy, Linda Charmaraman, Ineke Ceder, Amanda Richer, and Wendy Surr Out-of-school time (OST) youth programs are inherently difﬁcult to assess. They are often very dynamic: Many youth interact with one another and with staff members in various physical environments. Despite the challenge, measuring quality is critical to help program directors and policy makers identify where to improve and how to support those improvements. This article describes recent research on the Assessment of Program Practices Tool (APT), establishing its strength as an evaluation and tracking tool for OST programs. Funded by the William T. Grant Foundation and Virginia B. Toulmin Foundation, the validation was conducted in two phases. The ﬁrst phase was designed to evaluate the scientiﬁc rigor of the tool. Based on the ﬁndings from the ﬁrst phase, the second aimed to inform improvements in the tool and its training. Our testing so far shows that online video-based training needs to be more specialized in order to improve rating reliability for high-stakes users, such as third-party evaluators. ALLISON TRACY has over 15 years of experience providing methodological and statistical consultation for researchers in a wide variety of disciplines, topics, and institutions. She has worked on the psychometric analysis of A Program Assessment System (APAS) and its components, including the Assessment of Program Practices Tool (APT) . LINDA CHARMARAMAN, PhD, co-principal investigator of the APT Validation Study II, is a research scientist at the Wellesley Centers for Women specializing in positive youth development. She has conducted research and evaluations of in-school and out-of-school time (OST) programs for over 12 years. INEKE CEDER is a research associate at the Wellesley Centers for Women, where she has been involved with projects on child and adolescent development, sexual education, and women’s leadership. She was the data collection manager and survey developer for Phase 2 of the APT Validation Study. AMANDA RICHER is a research associate for the National Institute on Out-of-School Time and assistant methodologist for the Wellesley Centers for Women. She has been involved in psychometric testing of OST assessments and she has supported research in youth development. WENDY SURR, Senior Researcher at the American Institutes for Research, has more than 30 years of experience leading research and evaluation studies and other educational initiatives. While at NIOST, she co-created APAS, including developing the APT and a set of teacher, staff, and youth survey instruments for measuring student nonacademic outcomes. She served as co-principal investigator and project director for Phase 1 of the APT Validation Study. Tracy, Charmaraman, Ceder, Richer, & Surr MEASURING PROGRAM QUALITY 3 The Assessment of Program Practices Tool The APT is an observational instrument created in 2005 by Beth Miller and Wendy Surr of the National Institute on Out-of-School Time (NIOST) to measure OST process quality: observable aspects of a program in action. Research suggests that process quality contributes to the 21st century skills, attitudes, and behaviors youth need tobe successful in school and the workplace (Miller, 2005) . The APT is one component of A Program Assessment System (APAS), an integrated quality and outcome assessment system developed by NIOST in partnership with the Massachusetts Department of Elementary and Secondary Education 21st Century Community Learning Centers (21st CCLC) initiative. Currently, the APT is used in 33 states and in Canada by over 1,500 individuals and 600 OST programs. The APT is designed to support program self-assessment and improvement efforts. Increasingly, it is also being used by external stakeholders, such as funders and sponsors of afterschool programs, to ensure that programs

FAQs

Why is measuring quality in out-of-school time (OST) programs challenging?

OST programs are inherently dynamic, with many youth interacting with each other and staff across different physical environments, making assessment harder than in more controlled settings.

What is the Assessment of Program Practices Tool (APT) used for?

The APT measures OST process quality by capturing observable aspects of programs in action and supports program self-assessment, improvement, and external quality rating.

What does Phase 1 of the APT Validation Study evaluate?

Phase 1 evaluates the tool’s technical properties and scientific rigor using tests such as internal consistency, test-retest stability, interrater reliability, and predictive validity.

Oct 15, 2026

Uploaded by Stanley