Measuring Program Quality, Part 2 - Addressing Potential Cultural Bias in a Rater Reliability Exam

Oct 15, 2026

Uploaded by Stanley

Measuring Program Quality, Part 2 examines how program-quality assessment tools must avoid cultural bias, especially in rater reliability exams used to judge observation accuracy. The discussion links culturally biased rater testing to fairness and to downstream youth outcomes, where untrained raters may assign systematically negative scores. It reviews sources of rater bias, including personal interpretations and expectations, and outlines strategies such as explicit anchors and master-scored video training to improve rater accuracy and reduce reliability “drift.”

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages9

Reading Time23 min

AccessFree to download

Measuring Program Quality, Part 2

Addressing Potential Cultural Bias in a Rater Reliability Exam

Rater Reliability and Rater Bias

Training and Practice to Minimize Bias

Explicit Rating Anchors and Master-Scored Video Training

NEW FROM NIOST Measuring Program Quality, Part 2 Addressing Potential Cultural Bias in a Rater Reliability Exam Amanda Richer, Linda Charmaraman, and Ineke Ceder Like instruments used in afterschool programs to assess children’s social and emotional growth or to evaluate staff members’ performance, instruments used to evaluate program quality should be free from bias. Practitioners and researchers alike want to know that assessment instruments, whatever their type or intent, treat all people fairly and do not privilege people from certain groups over others. In the case of observation instruments, concern about bias extends beyond the instrument itself to the people doing the observation: how they apply the instrument’s rubrics or standards in specific afterschool settings. A vital subset of concern about possible rater bias is whether any exam used to assess rater reliability itself carries unintended bias toward some groups of people. This issue is not only a matter of fairness. Culturally biased rater reliability testing can directly affect youth outcomes. For example, an urban youth program in a low-income neighborhood with many people of color could receive negative scores from a rater who was not trained and certified to overcome any implicit biases related to racial and cultural practices different from his AMANDA RICHER is a research associate and associate methodologist for the National Institute on Out-of-School Time and the Wellesley Centers for Women. She has been involved in psychometric testing of the APT since the first validation study and continues to use data to enhance OST tools. LINDA CHARMARAMAN, PhD, is a research scientist at the Wellesley Centers for Women who has served as principal or coprincipal investigator of two APT validation studies. She is engaged in using research evidence to inform policy and practice and in reducing structural inequality gaps in research. INEKE CEDER is a research associate at the Wellesley Centers for Women whose work focuses on gender and race, women’s leadership, and sex education. Her involvement on all three validity studies of the APT has driven home that research data can drive social change and bring equity for all. or her own. When observation of what each point on the rating ratings affect funder decisions, the scale looks like, so that raters can problem becomes acute. Culturally biased rater clearly see what constitutes a rating Overcoming this possible reliability testing can of, for example, 1, 2, 3, or 4. Rater source of bias is our concern in directly affect youth training that uses videos with athiseasrarchticleandasemembers ofvaluation tetahme refor- outcomes. For example, an reahasl-wobeerlndsexamhownpletoofieachmprovaencrhatoerr the Assessment of Program Prac- urban youth program in a accuracy (Kishida & Kemp, 2010; tices Tool (APT) . As we conducted low-income neighborhood Schlientz et al. , 2009) . studies to establish the scientific with many people of color Another strategy to reduce bias validity of APT (described in Tra- could receive negative is master scoring of video clips to cy&, SChurar,rm20ara16m),anwe, Cederuncov,eRicredhaerp-, scores from a rater who esIntablithisshstraate“ggoy,ldhistanghlydtardrain”esdcoraned. parent cultural bias in the prelimi- was not trained and experienced raters, usually working nary results of the APT rater reli- certified to overcome any in groups of two or three, all rate atboilityachiexaevemthse: Whtargite rateet ratersoftendedagree- implicit biases related to thaned samediscusvsiduenostil ,thceoympacanreagnoreeteosn, ment with master scores more of- racial and cultural practices a single master score for each video. ten than Black raters. This article different from his or her Use of master-scored video training describes the follow-up study we own. When observation improves rater accuracy and miticonducted to address the sources ratings affect funder gates rater “drift” (Bell et al.

FAQs

Why should program quality assessment instruments be free from bias?

Assessment tools should treat all people fairly and not privilege groups over others. Bias can also affect youth outcomes when rater reliability testing leads to negative scores.

What are common sources of rater bias in observation-based assessments?

Rater bias can stem from raters’ personal interpretations of the scale and their expectations. Training and practice can reduce these biases by improving accuracy and consistency.

How can training reduce cultural bias in rater reliability exams?

The document highlights strategies such as explicit rating anchors and master-scored video clips. Master-scored training can improve rater accuracy and help reduce reliability “drift,” supporting more equitable ratings.

Oct 15, 2026

Uploaded by Stanley