PRINCIPLES AND PRACTICE OF ON-DEMAND TESTING - January 2009

Oct 20, 2026

Uploaded by Sarah

Research commissioned by Ofqual reviews how computer-technology advances enable on-demand testing in the UK and examines implications for high-stakes general qualifications. The literature review covers the scope of on-demand testing, from more frequent test windows to anytime, anywhere delivery, highlighting benefits such as efficiency, timely results, and scheduling flexibility, alongside system-wide risks. Stakeholder findings from teachers, students, and examiners address concerns about exam pressure and integrity, and the role of pre-testing and rigorous models.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages68

Reading Time2 h 51 min

AccessFree to download

A. Executive Summary

B. Introduction

C. Literature Review

1. An on-demand world

2. What is on-demand testing?

3. The vision of on-demand: major drivers behind on-demand testing

4. Barriers to on-demand testing

5. Beyond linear testing

D. Stakeholder Perspectives on On-Demand Testing

1. Introduction

2. Method

3. Student findings: Dominant themes

4. Teacher findings: Dominant themes

5. Summary and conclusions from teachers and students

E. The Providers’ Perspective

1. Introduction

2. Profiles of organisations

3. Current practice in on-demand testing

F. Principles of On-Demand Testing

1. Examination standards

2. Accessibility

3. The burden of assessment

4. Communication

G. Conclusion

H. References

PRINCIPLES AND PRACTICE OF ON-DEMAND TESTING Christopher Wheadon, Claire Whitehouse, Victoria Spalding, Katherine Tremain & Melody Charman January 2009 Copyright © 2008 AQA and its licensors The Assessment and Qualifications Alliance (AQA) is a company limited by guarantee registered in England and Wales (company number 3644723) and a registered charity (registered charity number 1073334) . Registered address: AQA, Devas Street, Manchester M15 6EX. Dr Michael Cresswell, Director General. Table of Contents Page A. Executive Summary 4 B. Introduction 7 C. Literature Review 8 1. An on-demand world 8 2. What is on-demand testing? 8 3. The vision of on-demand: major drivers behind on-demand testing 10 3.1 Efficiency 10 3.2 Bridging the gap with formative assessment 12 3.3 Validity 13 3.4 Competitive advantage 14 4. Barriers to on-demand testing 15 4.1 Technological infrastructure 15 4.2 Maintaining standards 15 4.2.1 Item Response Theory 15 4.2.2 Violations of the IRT model 16 4.2.3 Test equating models for on-demand testing 18 4.2.4 Evaluating the test equating designs 19 4.2.5 The National Assessment of Educational Progress (NAEP) anomaly: A cautionary tale 20 4.2.6 Monitoring outcomes 20 4.3 The skills deficit 21 4.4 Equity: The digital divide 21 4.5 Test construction models 22 5. Beyond linear testing 22 5.1 Computer Adaptive Testing 22 5.2 Multistage testing 23 6. Conclusion 25 D. Stakeholder Perspectives on On-Demand Testing 26 1. Introduction 26 2. Method 26 2.1 Participants 26 2.2 Procedure 27 3. Student findings: Dominant themes 27 3.1 Theme 1: Exam pressure 27 3.2 Theme 2: Exam integrity 29 3.3 Theme 3: Frequency of exams 30 3.4 Theme 4: The effect of on-demand testing on schools 31 3.5 Theme 5: The effect of learning factors on on-demand testing 32 4. Teacher findings: Dominant themes 33 4.1 Theme 1: Exam integrity 33 4.2 Theme 2: The effect of on-demand testing on schools 34 4.3 Theme 3: Requirements for on-demand testing 35 4.4 Theme 4: The effect of learning factors on on-demand testing 36 5. Summary and conclusions from teachers and students 37 Page 6. Examiner findings: Dominant themes 37 6.1 Theme 1: The importance of pre-testing 37 6.2 Theme 2: Implications of on-demand testing for item re-use, release and security 38 6.3 Theme 3: Other implications of on-demand testing 39 7. Summary and conclusions from examiners 39 E. The Providers’ Perspective 40 1. Introduction 40 2. Profiles of organisations 40 2.1 Universities Medical Assessment Partnership (UMAP) 40 2.2 City & Guilds 42 2.3 Scottish Qualifications Authority (SQA) 42 2.4 Ifs School of Finance 43 2.5 CEM Centre 43 2.6 EDI 43 2.7 DSA 44 3. Current practice in on-demand testing 45 3.1 The infrastructure of on-demand testing 45 3.2 Standard setting and maintenance 45 3.3 Items and tests 47 3.4 Technology 49 3.5 Security 49 4. Conclusions 49 F. Principles of On-Demand Testing 50 1. Examination standards 50 2. Accessibility 50 3. The burden of assessment 50 4. Communication 51 G. Conclusion 52 H. References 54 Appendices 58 A. EXECUTIVE SUMMARY This research was commissioned by Ofqual to review how advances in computer technology were enabling on-demand testing in the UK and to consider the implications of these advances for high stakes general qualifications. Their intention was to deepen their understanding of the concerns of stakeholders in this area by looking at current practice in the UK and abroad. The findings will inform Ofqual's regulatory approach to on-demand testing. The first section of this report is a review of the literature relevant to on-demand testing. This review suggests that on-demand testing ranges from the provision of more frequent test windows to anytime, anywhere testing. In its purest form, on-demand testing clearly supports the personalisation policy agenda and the desire

FAQs

What was the purpose of this research commissioned by Ofqual?

To review how advances in computer technology enable on-demand testing in the UK and to consider implications for high-stakes general qualifications, informing Ofqual’s regulatory approach.

What does the report describe as the range of on-demand testing?

On-demand testing ranges from providing more frequent test windows to anytime, anywhere testing, supporting personalisation in its purest form and delivering efficiency and scheduling flexibility in broader forms.

What concerns did teachers and students raise about on-demand testing?

They were sceptical that pure on-demand testing would support personalised learning without greater teacher support, and they worried it could increase exam pressure via competition among peers and parents.

Oct 20, 2026

Uploaded by Sarah