AWARDING OBJECTIVE TEST PAPERS: IS THERE A CORRECT ANSWER? - SUMMARY

Oct 20, 2026

Uploaded by Sarah

This paper reviews the main methods used to award objective tests (OTs), focusing on approaches that incorporate professional judgement rather than relying solely on back calculation or predicted outcomes. It contrasts existing AQA practices, including earlier trials of the Bookmark method, with Angoff-based approaches and other available techniques. The discussion compares pros and cons across absolute, consensus, and bookmarking methods, and considers how item response theory (IRT) data can support judges. It also frames how a new Diploma OT unit should be awarded in January 2009.

LanguageEnglish

CategoryExam

FormatPDF

Pages43

Reading Time1 h 48 min

AccessFree to download

1. Summary

2. Background: Existing AQA Approaches to Awarding Objective Test Components

2.1 Linear GCSE: Back calculation

2.2 Two-component GCE units: The Angoff method

2.3 Single component GCSE Science modules and Functional Skills

3. The Methods Available

3.1 Absolute methods

3.2 Bookmarking

3.3 Compromise methods

3.4 The Direct Consensus method

4. Pros and Cons

4.4 Boookmark-based versus Angoff-based approaches

4.5 Using IRT data to assist judges in making Angoff judgements

5. Discussion

6. References

AWARDING OBJECTIVE TEST PAPERS: IS THERE A CORRECT ANSWER? CONTENTS Lesley Meyer 1. SUMMARY………………………………………………………………………………………. 1 2. BACKGROUND: EXISTING AQA APPROACHES TO AWARDING OBJECTIVE TEST COMPONENTS………………………………………………………………………………..... 1 2.1 Linear GCSE: Back calculation………………………………………………………………… 1 2.2 Two-component GCE units: The Angoff method…………………………………………….. 2 2.3 Single component GCSE Science modules and Functional Skills: Common candidates and common centres’ data and/or KS3 predicted outcomes………………………………. 3 3. THE METHODS AVAILABLE…………………………………………………………………. 4 3.1 Absolute methods……………………………………………………………………………….. 4 3.1.1 The Nedelsky method…………………………………………………………………… 5 3.1.2 The Ebel method…………………………………………………………………………. 6 3.1.3 The Angoff method………………………………………………………………………. 7 a) The ‘Modified Angoff’method…………………………………………………….... 8 b) The ‘Yes/No’ method………………………………………………………………... 9 c) The ‘Extended Angoff’method…………………………………………………….. 10 d) General discussion of the Angoff approaches…………………………………… 10 3.1.4 Jaegar’s method………………………………………………………………………….. 12 3.2 Bookmarking……………………………………………………………………………………... 13 3.2.1 The Bookmark method…………………………………………………………………... 13 3.2.2 Item mapping……………………………………………………………………………... 16 3.3 Compromise methods…………………………………………………………………………... 17 3.3.1 Hofstee’s method……………………………………………………………………….... 17 3.3.2 Beuk’s method……………………………………………………………………………. 18 3.4 The Direct Consensus method………………………………………………………………… 20 4. PROS AND CONS……………………………………………………………………………… 21 4.1 The Absolute methods………………………………………………………………………….. 22 4.2 The Direct Consensus method………………………………………………………………… 22 4.3 The Bookmark method………………………………………………………………………….. 23 4.4 Boookmark-based versus Angoff-based approaches……………………………………….. 23 4.5 Using IRT data to assist judges in making Angoff judgements…………………………….. 25 4.5.1 Presenting item performance information for every total mark on the test………… 25 4.5.2 Feeding back to the judges on the accuracy of their probability ratings…………… 27 4.6 AQA’s experience of the Bookmarking method……………………………………………… 27 4.7 Is preserving the item sequence an issue?....................................................................... 30 5. DISCUSSION……………………………………………………………………………………. 30 6. REFERENCES…………………………………………………………………………………... 36 APPENDIX A…………………………………………………………………………………….. 39 APPENDIX B…………………………………………………………………………………….. 40 APPENDIX C…………………………………………………………………………………….. 41 1. SUMMARY In AQA, Objective Tests (OTs) have, in GCSEs generally, historically been awarded via a process known as ‘back calculation’ from other components (see §2 . 1) . More recently, other methods such as Angoff (GCE) and KS3 predicted outcomes/IRT (GCSE Science) have also been employed. However, back calculation and the use of predicted outcomes do not involve professional judgement. This is in contrast to the Angoff approach, but this latter method has had mixed success as used in the current AQA GCEs. In 2004 and 2005, trials of awarding OTs using the Bookmark method were carried out in the AQA Manchester Research Department (Fowles, 2004 & 2005), but these did not give promising results and the approach was not pursued. One OT unit (comprising one component – entirely OT) in the new Diploma qualifications is being awarded in January 2009, prompting the need for discussion on the most appropriate way to award this unit specifically, but also consequently for a review of the current methods available of awarding OTs generally to fully inform that decision. In the wider view, the awarding of OTs is gathering an increasingly high profile within Ofqual. This autumn (2008) the Standards and Technical Advisory Group (STAG) awarding subgroup is intending to draft a procedural paragraph for the JCQ Setting Standards papers1 on the awarding of OTs, and Edexcel and OCR will be awarding new OT components in the Diploma in the forthcoming series

FAQs

Why is awarding objective test (OT) papers challenging in AQA?

Because there is limited evidence on a completed script beyond the number of correctly answered questions, a judgemental scrutiny process used for written papers is not appropriate. OT awarding therefore requires alternative methods.

What is the difference between back calculation and Angoff-based approaches?

Back calculation and predicted outcomes are not based on professional judgement, whereas Angoff approaches are designed to include judgements from experts. The paper notes mixed success with Angoff in current AQA GCEs.

How does the Bookmarking method fit into AQA’s experience?

Trials using the Bookmark method were carried out in the AQA Manchester Research Department, but results were not promising and the approach was not pursued further at that time.

Oct 20, 2026

Uploaded by Sarah