Framework for Considering Device and Interface Features That May Affect Student Performance on the National Assessment of Educational Progress

Sep 4, 2026

A framework document examines how student performance in the National Assessment of Educational Progress may be influenced by device and interface features. It reviews NAEP validity studies and relevant literature, drawing on mode/device research and cognitive laboratory findings. The report synthesizes interdisciplinary foundations using cognitive load theory, human-computer interaction, resource activation, and construct validity. It then organizes framework considerations for writing and nonwriting subjects, including usability, familiarity, and multidimensional application to NAEP, followed by research directions.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages60

Reading Time2 h 31 min

AccessFree to download

Introduction

Comparability Literature

Mode/Device Studies

Cognitive Laboratories

The Need for a Framework

Interdisciplinary Foundations

Cognitive Load Theory and Human-Computer Interaction

The Applicability of CLT to Assessment

Resource Activation and Construct Validity

Framework Considerations: Writing

A Cognitive Model of Writing

Text Production, Typing Skills, and Cognitive Load

Text Production and Device Form Factors

The Coordinated Activities of Writing and Reading

Reading and Device Form Factors

Revision: The Impact of Devices, Interfaces, and Computer Skills

Motivation and Device Performance

Framework Considerations: Nonwriting Subjects

Visual Stimuli

Calculators

Math Markings: Annotation

Math Markings: Equations and Symbols

Tools, Animation, and Interactivity

The Proposed Framework for Considering Device and Interface Features

Usability and Familiarity

Framework Multidimensionality

Applying the Framework to NAEP

Research Directions for NAEP: Device/Interface

A Framework for Considering Device and Interface Features That May Affect Student Performance on the National Assessment of Educational Progress Denny Way College Board Ellen Strain-Seymour Pearson March 2021 Commissioned by the NAEP Validity Studies (NVS) Panel The NAEP Validity Studies (NVS) Panel was formed in 1995 to provide a technical review of NAEP plans and products and to identify technical concerns and promising techniques worthy of further study and research. The members of the panel have been charged with writing focused studies and issue papers on the most salient of the identified issues. Panel Members: Keena Arbuthnot Louisiana State University Peter Behuniak Criterion Consulting, LLC Jack Buckley American Institutes for Research James R. Chromy Research Triangle Institute (retired) Phil Daro Strategic Education Research Partnership (SERP) Institute Richard P. Durán University of California, Santa Barbara David Grissmer University of Virginia Larry Hedges Northwestern University Gerunda Hughes Howard University Ina V.S. Mullis Boston College Scott Norton Council of ChiefState School Officers James Pellegrino University ofIllinois at Chicago Gary Phillips American Institutes for Research Lorrie Shepard University of Colorado Boulder David Thissen University of North Carolina, Chapel Hill Gerald Tindal University of Oregon Sheila Valencia University of Washington Denny Way College Board Project Director: Sami Kitmitto American Institutes for Research Project Officer: Grady Wilburn National Centerfor Education Statistics For Information: NAEP Validity Studies (NVS) Panel American Institutes for Research 2800 Campus Drive, Suite 200 San Mateo, CA 94403 Email: [skitmitto@air.org](skitmitto@air.org) INTRODUCTION.....................................................................................................................................................................1 COMPARABILITY LITERATURE ..........................................................................................................................................4 Mode/Device Studies ......................................................................................................................................... 4 Cognitive Laboratories ....................................................................................................................................... 5 The Need for a Framework ................................................................................................................................ 7 INTERDISCIPLINARY FOUNDATIONS.................................................................................................................................9 Cognitive Load Theory and Human-Computer Interaction................................................................................... 9 The Applicability of CLT to Assessment............................................................................................................ 11 Resource Activation and Construct Validity ...................................................................................................... 12 FRAMEWORK CONSIDERATIONS: WRITING ..................................................................................................................14 A Cognitive Model of Writing ............................................................................................................................ 14 Text Production, Typing Skills, and Cognitive Load .......................................................................................... 14 Text Production and Device Form Factors........................................................................................................ 15 The Coordinated Activities of Writing and Reading ........................................................................................... 17 Re

FAQs

What is the purpose of the NAEP Validity Studies (NVS) Panel in this report?

The panel provides technical review of NAEP plans and products, identifies technical concerns and promising techniques, and commissions focused studies and issue papers on the most salient issues.

Which theoretical foundations does the framework rely on?

The report integrates cognitive load theory and human-computer interaction, plus connections to assessment through applicability of CLT, resource activation, and construct validity.

How does the report structure framework considerations across subjects?

It separates considerations for writing and nonwriting subjects, covering device/interface effects on writing processes and on elements such as visual stimuli, calculators, and math notations and tools.

Sep 4, 2026

Upload by Genevieve