[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"detail-sidebar-cat-0-en-105":3,"doc-detail-528572-en":59,"doc-seo-528572-105":145},{"code":4,"msg":5,"data":6},0,"success",[7,13,18,23,28,33,38,43,48,51,55],{"id":8,"doc_module":4,"doc_module_name":9,"category_name":10,"show_sort_weight":11,"slug":12},1,"Document","Story & Novel",90,"story-novel",{"id":14,"doc_module":4,"doc_module_name":9,"category_name":15,"show_sort_weight":16,"slug":17},2,"Literature",80,"literature",{"id":19,"doc_module":4,"doc_module_name":9,"category_name":20,"show_sort_weight":21,"slug":22},4,"Exam",70,"exam",{"id":24,"doc_module":4,"doc_module_name":9,"category_name":25,"show_sort_weight":26,"slug":27},5,"Comic",60,"comic",{"id":29,"doc_module":4,"doc_module_name":9,"category_name":30,"show_sort_weight":31,"slug":32},6,"Technology",50,"technology",{"id":34,"doc_module":4,"doc_module_name":9,"category_name":35,"show_sort_weight":36,"slug":37},7,"Healthcare",40,"healthcare",{"id":39,"doc_module":4,"doc_module_name":9,"category_name":40,"show_sort_weight":41,"slug":42},8,"Research & Report",30,"research-report",{"id":44,"doc_module":4,"doc_module_name":9,"category_name":45,"show_sort_weight":46,"slug":47},9,"Religion & Spirituality",20,"religion-spirituality",{"id":46,"doc_module":4,"doc_module_name":9,"category_name":49,"show_sort_weight":46,"slug":50},"World Cup","world-cup",{"id":52,"doc_module":4,"doc_module_name":9,"category_name":53,"show_sort_weight":52,"slug":54},10,"Lifestyle","lifestyle",{"id":56,"doc_module":4,"doc_module_name":9,"category_name":57,"show_sort_weight":24,"slug":58},19,"General","general",{"code":4,"msg":5,"data":60},{"doc_id":61,"user_id":62,"nickname":63,"user_avatar":64,"doc_module":4,"category_id":39,"category_name":40,"doc_title":65,"doc_description":66,"doc_content":67,"file_id":68,"file_url":69,"file_type":70,"file_size":71,"view_count":29,"is_deleted":4,"is_public":8,"is_downloadable":8,"audit_status":8,"page_count":72,"language":73,"language_code":74,"site_id":75,"html_lang":74,"table_of_contents":76,"faqs":77,"seo_title":78,"seo_description":66,"update_tm":79,"read_time":80,"preview_page_count":81,"preview_urls":82},528572,962085564549,"Aditya","https://ap-avatar.wpscdn.com/davatar_085a072bc5b1113ac321206ff7593b45","Reliability Issues With Performance Assessments - A Collection of Papers","A collection of six research papers examines reliability in performance assessment and performance testing, focusing on how measurement decisions depend on reliability information. It addresses definitions and uses of reliability beyond rater agreement, covering reliability estimates from conditional standard errors, polytomous IRT performance levels, and bootstrapping. Additional chapters evaluate measurement precision with multiple forms, raters, and tasks, propose weights under a congeneric model, and discuss reliability issues and possible solutions, emphasizing decision consistency and technical guidance for large-scale, high-stakes contexts.","# Reliability Issues With PerformanceAssessments:A Collection of Papers\n\nDean A.ColtonXiaohong GaoDeborah J.HarrisMichael J.Kolen  \nDara Martinovich-BarhiteTianyou WangCatherine J.Welch  \nFor additional copies write:ACT Research Report SeriesPO Box 168lowa City,Iowa 52243-0168  \nC1997 by ACT,Inc.All rights reserved.  \nReliability Issues With Performance Assessments:A Collection of Papers  \nDean A.ColtonXiaohong GaoDeborah J.HarrisMichael J.KolenDara Martinovich-BarhiteTianyou WangCatherine J.Welch  \nTable of Contents  \nPage  \nIntroduction………………………………………………………………………………………………………………………………………………………iii  \nUsing Reliabilities to Make Decisions…………………………………………………………………………………………………1  \nDeborah J.Harris  \nConditional Standard Errors, Reliability, and Decision Consistency  \nPerformance Levels Using Polytomous IRT……………………………………………………………………………13  \nTianyou Wang, Michael J. Kolen, Deborah J.Harris  \nAssessing the Reliability of Performance Level Scores Using Bootstrapping…………41  \nDean A. Colton, Xiaohong Gao, Michael J. Kolen  \nEvaluating Measurement Precision of Performance Assessment  \nWith Multiple Forms, Raters, and Tasks………………………………………………………………………………57  \nXiaohong Gao, Dean A.Colton  \nWeights That Maximize Reliability Under a Congeneric Model  \nfor Performance Assessment………………………………………………………………………………………………………………………77  \nTianyou Wang  \nReliability Issues and Possible Solutions……………………………………………………………………………95  \nCatherine J. Welch,Dara Martinovich-Barhite  \n## Introduction\n\nThis report consists of six papers,each dealing with some aspect of reliability andperformance testing.One of the papers,Welch and Martinovich-Barhite,was presented atthe 1997 Annual Meeting of the American Educational Research Association in asymposium called Issues in Large-Scale Portfolio Assessment.Versions of the other fivepapers were presented at the 1996 Annual Meeting of the American Educational ResearchAssociation as part of a symposium called Technical Issues Involving Reliability andPerformance Assessments.The authors would like to thank the discussants of the twosymposia,Ed Wolfe,and Robert L.Brennan and Nancy L.Allen,respectively,for theircomments during the two sessions,and Bradley A.Hanson and E.Matthew Schulz fortheir comments on a draft report.  \n## Using Reliabilities to Make Decisions\n\nDeborah J.Harris  \n2  \nAbstract  \nFor a variety of reasons,there has been an increased use of performanceassessments in high stakes and/or large scale situations.There is a long history of usingperformance assessments for classroom measurement;however,using these types ofassessments beyond a single classroom(where a single administration has more longterm consequences than whether to reteach the previous day's lesson)leads to anincreased need for valid,reliable assessments.Validity and reliability issues relating toperformance assessments have been much discussed,but further research and technicaldevelopment is needed.For example,reliability with performance assessments hasfrequently been relegated to solely the agreement among the raters scoring theassessments.Although this is certainly an important component,it is not sufficient toensure a reliable assessment.  \nThis paper addresses the use of reliability information,such as that provided inthe later papers in this report,in decision making.Specifically,choosing a score scale,forming a composite score,choosing a cut score,selecting a test,and similar issues arebriefly discussed.Making a decision by choosing the highest reliability estimate does notalways appear to be the optimal decision,particularly when reliability can be assessed indifferent ways(e.g.,rater agreement,generalizability coefficient,using a theoreticalmodel,or bootstrapping),and when the typical reliability estimate used withperformance assessments,rater agreement,may not be the most relevant,given thepurpose of the assessment.  \nThe author would like to thank Michael J.Kolen and Catherine J.Welch for theircomments on an earlier draft.  \n3  \n### Using reli","cbCaienlz6hdTgGv","https://ap.wps.com/l/cbCaienlz6hdTgGv","pdf",4353265,142,"English","en",105,"# Introduction\n# Using Reliabilities to Make Decisions\n# Conditional Standard Errors, Reliability, and Decision Consistency\n# Performance Levels Using Polytomous IRT\n# Assessing the Reliability of Performance Level Scores Using Bootstrapping\n# Evaluating Measurement Precision of Performance Assessment With Multiple Forms, Raters, and Tasks\n# Weights That Maximize Reliability Under a Congeneric Model for Performance Assessment\n# Reliability Issues and Possible Solutions","[{\"question\":\"Why is reliability especially important in large-scale or high-stakes performance assessments?\",\"answer\":\"Because extending performance assessments beyond a single classroom increases the need for assessments that remain valid and reliable under consequential outcomes. Reliability supports making defensible decisions when the assessment is used at scale.\"},{\"question\":\"How does the report frame reliability beyond rater agreement?\",\"answer\":\"It notes that rater agreement is important but not sufficient to ensure a reliable assessment. The report emphasizes using reliability information in decision making and considering reliability assessed in multiple ways, such as generalizability and bootstrapping.\"},{\"question\":\"What kinds of methods are presented to estimate reliability for performance assessments?\",\"answer\":\"The papers cover conditional standard errors and decision consistency, polytomous IRT for performance levels, bootstrapping for reliability of level scores, and precision evaluation across multiple forms, raters, and tasks. It also includes maximizing reliability under a congeneric model.\"}]","Reliability Issues With Performance Assessments - A Collection of Papers | PDF",1791519120,358,15,[83,87,91,96,100,104,108,112,116,120,123,128,133,137,142],{"page_no":8,"url":84,"width":85,"height":86},"https://res-academy.cache.wpscdn.com/upload/kdocshare/2026-11-17/0e15a5099334ba0642816d77d48fbfc8.webp",983,1330,{"page_no":14,"url":88,"width":89,"height":90},"https://res-academy.cache.wpscdn.com/upload/kdocshare/2026-11-17/dc61b48d0758d3fd8091538c6b8f8a26.webp",982,1272,{"page_no":92,"url":93,"width":94,"height":95},3,"https://res-academy.cache.wpscdn.com/upload/kdocshare/2026-11-17/fa1c3d308e8a84f4e50f8efc75760237.webp",981,1312,{"page_no":19,"url":97,"width":98,"height":99},"https://res-academy.cache.wpscdn.com/upload/kdocshare/2026-11-17/564961edd6d4899577da0ebd32fc9ff4.webp",980,1311,{"page_no":24,"url":101,"width":102,"height":103},"https://res-academy.cache.wpscdn.com/upload/kdocshare/2026-11-17/cf7d2fbfc66483351bc314407e4014da.webp",979,1309,{"page_no":29,"url":105,"width":106,"height":107},"https://res-academy.cache.wpscdn.com/upload/kdocshare/2026-11-17/6b9394c3b3625d5f37d4e29327de22ec.webp",978,1308,{"page_no":34,"url":109,"width":110,"height":111},"https://res-academy.cache.wpscdn.com/upload/kdocshare/2026-11-17/467bd39e9cdab634622a6df9a25e0b29.webp",977,1307,{"page_no":39,"url":113,"width":114,"height":115},"https://res-academy.cache.wpscdn.com/upload/kdocshare/2026-11-17/d8c0f266781c9c2bd2dc639113846dad.webp",976,1306,{"page_no":44,"url":117,"width":118,"height":119},"https://res-academy.cache.wpscdn.com/upload/kdocshare/2026-11-17/ca1ff5db508019b3c810f339a0745394.webp",975,1302,{"page_no":52,"url":121,"width":122,"height":119},"https://res-academy.cache.wpscdn.com/upload/kdocshare/2026-11-17/080d0c838dc13d4976551f64fb2cae67.webp",974,{"page_no":124,"url":125,"width":126,"height":127},11,"https://res-academy.cache.wpscdn.com/upload/kdocshare/2026-11-17/ceaa89e76f89b23226d2a37b0b7351d7.webp",973,1296,{"page_no":129,"url":130,"width":131,"height":132},12,"https://res-academy.cache.wpscdn.com/upload/kdocshare/2026-11-17/63a1fc955be5c6255dd89b6da5538d91.webp",972,1295,{"page_no":134,"url":135,"width":136,"height":132},13,"https://res-academy.cache.wpscdn.com/upload/kdocshare/2026-11-17/7462ab4668d8112da1ccab57796a1baf.webp",971,{"page_no":138,"url":139,"width":140,"height":141},14,"https://res-academy.cache.wpscdn.com/upload/kdocshare/2026-11-17/3c1fe57e63194213bddc3f1877f841f4.webp",970,1294,{"page_no":81,"url":143,"width":144,"height":141},"https://res-academy.cache.wpscdn.com/upload/kdocshare/2026-11-17/c9f0aae7be9ed653b2f83ebf643ff484.webp",969,{"code":4,"msg":146,"data":147},"ok",{"site_id":75,"language":74,"slug":148,"title":65,"keywords":149,"description":66,"schema_data":150,"social_meta":203,"head_meta":205,"extra_data":207,"updated_unix":208},"reliability-issues-with-performance-assessments-a-collection-of-papers","",{"@graph":151,"@context":202},[152,165,185],{"@type":153,"itemListElement":154},"BreadcrumbList",[155,159,161,163],{"item":156,"name":157,"@type":158,"position":8},"https://docshare.wps.com","Home","ListItem",{"item":160,"name":9,"@type":158,"position":14},"https://docshare.wps.com/document/",{"item":162,"name":40,"@type":158,"position":92},"https://docshare.wps.com/document/research-report/",{"item":164,"name":65,"@type":158,"position":19},"https://docshare.wps.com/document/reliability-issues-with-performance-assessments-a-collection-of-papers/528572/",{"url":164,"name":65,"@type":166,"image":167,"author":172,"headline":65,"publisher":174,"fileFormat":177,"inLanguage":74,"description":66,"dateModified":178,"datePublished":179,"encodingFormat":177,"isAccessibleForFree":180,"interactionStatistic":181},"DigitalDocument",{"url":168,"@type":169,"width":170,"height":171},"https://docshare.wps.com/thumbnails/reliability-issues-with-performance-assessments-a-collection-of-papers/528572.png","ImageObject",300,407,{"name":63,"@type":173},"Person",{"url":156,"name":175,"@type":176},"DocShare","Organization","application/pdf","2026-11-09","2026-10-09",true,{"@type":182,"interactionType":183,"userInteractionCount":29},"InteractionCounter",{"@type":184},"ViewAction",{"@type":186,"mainEntity":187},"FAQPage",[188,194,198],{"name":189,"@type":190,"acceptedAnswer":191},"Why is reliability especially important in large-scale or high-stakes performance assessments?","Question",{"text":192,"@type":193},"Because extending performance assessments beyond a single classroom increases the need for assessments that remain valid and reliable under consequential outcomes. Reliability supports making defensible decisions when the assessment is used at scale.","Answer",{"name":195,"@type":190,"acceptedAnswer":196},"How does the report frame reliability beyond rater agreement?",{"text":197,"@type":193},"It notes that rater agreement is important but not sufficient to ensure a reliable assessment. The report emphasizes using reliability information in decision making and considering reliability assessed in multiple ways, such as generalizability and bootstrapping.",{"name":199,"@type":190,"acceptedAnswer":200},"What kinds of methods are presented to estimate reliability for performance assessments?",{"text":201,"@type":193},"The papers cover conditional standard errors and decision consistency, polytomous IRT for performance levels, bootstrapping for reliability of level scores, and precision evaluation across multiple forms, raters, and tasks. It also includes maximizing reliability under a congeneric model.","https://schema.org",{"og:url":164,"og:type":204,"og:title":65,"og:site_name":175,"og:description":66},"article",{"robots":206,"canonical":164},"index,follow",{"doc_id":61,"site_id":75},1791667273]