[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-seo-585383-105":3,"detail-sidebar-cat-0-en-105":81,"doc-detail-585383-en":130},{"code":4,"msg":5,"data":6},0,"ok",{"site_id":7,"language":8,"slug":9,"title":10,"keywords":11,"description":12,"schema_data":13,"social_meta":74,"head_meta":76,"extra_data":78,"updated_unix":80},105,"en","confusion-aware-rubric-optimization-for-llm-based-automated-grading","Confusion-Aware Rubric Optimization for LLM-based Automated Grading","","Accurate, unambiguous guidelines are essential for large language model (LLM) graders, yet manual prompt creation is often ineffective because LLMs may misread expert rubrics or miss needed domain detail. Existing optimization frameworks usually merge independent, unstructured error samples into one update, causing “rule dilution” that weakens grading logic. ConfusionAware Rubric Optimization (CARO) prevents guidance conflicts by separating error signals via confusion-matrix decomposition and applying mode-specific fixing patches with diversity-aware selection.",{"@graph":14,"@context":73},[15,34,56],{"@type":16,"itemListElement":17},"BreadcrumbList",[18,23,27,31],{"item":19,"name":20,"@type":21,"position":22},"https://docshare.wps.com","Home","ListItem",1,{"item":24,"name":25,"@type":21,"position":26},"https://docshare.wps.com/document/","Document",2,{"item":28,"name":29,"@type":21,"position":30},"https://docshare.wps.com/document/research-report/","Research & Report",3,{"item":32,"name":10,"@type":21,"position":33},"https://docshare.wps.com/document/confusion-aware-rubric-optimization-for-llm-based-automated-grading/585383/",4,{"url":32,"name":10,"@type":35,"image":36,"author":41,"headline":10,"publisher":44,"fileFormat":47,"inLanguage":8,"description":12,"dateModified":48,"datePublished":49,"encodingFormat":47,"isAccessibleForFree":50,"interactionStatistic":51},"DigitalDocument",{"url":37,"@type":38,"width":39,"height":40},"https://docshare.wps.com/thumbnails/confusion-aware-rubric-optimization-for-llm-based-automated-grading/585383.png","ImageObject",300,407,{"name":42,"@type":43},"Aria Callaghan","Person",{"url":19,"name":45,"@type":46},"DocShare","Organization","application/pdf","2026-10-30","2026-10-15",true,{"@type":52,"interactionType":53,"userInteractionCount":55},"InteractionCounter",{"@type":54},"ViewAction",8,{"@type":57,"mainEntity":58},"FAQPage",[59,65,69],{"name":60,"@type":61,"acceptedAnswer":62},"Why are accurate guidelines important for LLM-based automated grading?","Question",{"text":63,"@type":64},"LLM graders rely on prompt instructions and rubric operationalization to make decisions. If guidelines are ambiguous or poorly specified, the model can misinterpret expert intent and produce unreliable scores.","Answer",{"name":66,"@type":61,"acceptedAnswer":67},"What problem does “rule dilution” create in existing rubric optimization approaches?",{"text":68,"@type":64},"When frameworks aggregate independent and unstructured error samples into a single update step, conflicting constraints can weaken the model’s grading logic, reducing accuracy.",{"name":70,"@type":61,"acceptedAnswer":71},"How does CARO improve rubric optimization accuracy and efficiency?",{"text":72,"@type":64},"CARO decomposes monolithic error signals into distinct confusion-matrix modes, diagnoses and repairs specific misclassification patterns individually, and synthesizes targeted fixing patches selected with a diversity-aware mechanism to avoid guidance conflicts.","https://schema.org",{"og:url":32,"og:type":75,"og:title":10,"og:site_name":45,"og:description":12},"article",{"robots":77,"canonical":32},"index,follow",{"doc_id":79,"site_id":7},585383,1792192267,{"code":4,"msg":82,"data":83},"success",[84,88,92,96,101,106,111,114,119,122,126],{"id":22,"doc_module":4,"doc_module_name":25,"category_name":85,"show_sort_weight":86,"slug":87},"Story & Novel",90,"story-novel",{"id":26,"doc_module":4,"doc_module_name":25,"category_name":89,"show_sort_weight":90,"slug":91},"Literature",80,"literature",{"id":33,"doc_module":4,"doc_module_name":25,"category_name":93,"show_sort_weight":94,"slug":95},"Exam",70,"exam",{"id":97,"doc_module":4,"doc_module_name":25,"category_name":98,"show_sort_weight":99,"slug":100},5,"Comic",60,"comic",{"id":102,"doc_module":4,"doc_module_name":25,"category_name":103,"show_sort_weight":104,"slug":105},6,"Technology",50,"technology",{"id":107,"doc_module":4,"doc_module_name":25,"category_name":108,"show_sort_weight":109,"slug":110},7,"Healthcare",40,"healthcare",{"id":55,"doc_module":4,"doc_module_name":25,"category_name":29,"show_sort_weight":112,"slug":113},30,"research-report",{"id":115,"doc_module":4,"doc_module_name":25,"category_name":116,"show_sort_weight":117,"slug":118},9,"Religion & Spirituality",20,"religion-spirituality",{"id":117,"doc_module":4,"doc_module_name":25,"category_name":120,"show_sort_weight":117,"slug":121},"World Cup","world-cup",{"id":123,"doc_module":4,"doc_module_name":25,"category_name":124,"show_sort_weight":123,"slug":125},10,"Lifestyle","lifestyle",{"id":127,"doc_module":4,"doc_module_name":25,"category_name":128,"show_sort_weight":97,"slug":129},19,"General","general",{"code":4,"msg":82,"data":131},{"doc_id":79,"user_id":132,"nickname":42,"user_avatar":133,"doc_module":4,"category_id":55,"category_name":29,"doc_title":10,"doc_description":12,"doc_content":134,"file_id":135,"file_url":136,"file_type":137,"file_size":138,"view_count":55,"is_deleted":4,"is_public":22,"is_downloadable":22,"audit_status":22,"page_count":139,"language":140,"language_code":8,"site_id":7,"html_lang":8,"table_of_contents":141,"faqs":142,"seo_title":143,"seo_description":12,"update_tm":144,"read_time":112,"preview_page_count":4},962084926284,"https://ap-avatar.wpscdn.com/davatar_29158cc5080c5b710cf443261637dec0","Confusion-Aware Rubric Optimization for LLM-based  \nAutomated Grading  \nYucheng Chu Michigan State University [chuyuch2@msu.edu](chuyuch2@msu.edu)  \nYasemin Copur-Gencturk University of Southern  \nCalifornia [copurgen@usc.edu](copurgen@usc.edu)  \nHang Li Michigan State University  \n[lihang4@msu.edu](lihang4@msu.edu)[ ](lihang4@msu.edu)Joseph Krajcik  \nMichigan State University  \n[krajcik@msu.edu](krajcik@msu.edu)  \nKaiqi Yang  \nMichigan State University  \n[kqyang@msu.edu](kqyang@msu.edu)[ ](kqyang@msu.edu)Namsoo Shin  \nMichigan State University  \n[namsoo@msu.edu](namsoo@msu.edu)  \nJiliang Tang  \nMichigan State University  \n[tangjili@msu.edu](tangjili@msu.edu)  \nABSTRACT  \nAccurate and unambiguous guidelines are critical for large language model (LLM) based graders, yet manually crafting these prompts is often sub-optimal as LLMs can misinterpret expert guidelines or lack necessary domain specificity. Consequently, the field has moved toward automated prompt optimization to refine grading guidelines without the burden of manual trial and error. However, existing frameworks typically aggregate independent and unstructured error samples into a single update step, resulting in “rule dilution”where conflicting constraints weaken the model’s grading logic. To address these limitations, we introduce ConfusionAware Rubric Optimization (CARO), a novel framework that enhances accuracy and computational efficiency by structurally separating error signals. CARO leverages the confusion matrix to decompose monolithic error signals into distinct modes, allowing for the diagnosis and repair of specific misclassification patterns individually. By synthesizing targeted “fixing patches” for dominant error modes and employing a diversity-aware selection mechanism, the framework prevents guidance conflict and eliminates the need for resource-heavy nested refinement loops. Empirical evaluations on teacher education and STEM datasets demonstrate that CARO significantly outperforms existing SOTA methods. These results suggest that replacing mixed-error aggregation with surgical, mode-specific repair yields robust improvements in automated assessment scalability and precision.  \nKeywords  \nAutomated Grading, Large Language Models, Automated Prompt Optimization, Confusion Matrices.  \nYucheng Chu, Hang Li, Kaiqi Yang, Yasemin Copur-Gencturk, Joseph Krajcik, Namsoo Shin, and Jiliang Tang. Confusion-Aware Rubric Optimization for LLM-based Automated Grading. In Anthony Botelho, Maria Mercedes T. Rodrigo, Adish Singla, Hiroaki Ogata, Hyojeong So, and Young Hoan Cho (eds.) Proceedings of the 19th International Conference on Educational Data Mining, Seoul, Republic of Korea, June, 2026, pp. 67–78. International Educational Data Mining Society (2026) .  \n© 2026 Copyright is held by the author(s) . This work is distributed under the Creative Commons Attribution NonCommercial NoDerivatives 4.0 International (CC BY-NC-ND 4.0) license. [https://doi.org/10.5281/zenodo.21039998](https://doi.org/10.5281/zenodo.21039998)  \n1. INTRODUCTION  \nThe rapid shift toward personalized and competency-based learning has placed unprecedented challenges for traditional assessment frameworks. As personalized learning relies on understanding the specific needs for each learner, it necessitates high-quality formative assessment. In modern pedagogy, providing frequent and high-quality assessment feedback on students’ open-ended work is essential for diagnosing misconceptions and fostering deep conceptual understanding [2, 13, 25] . Yet these benefits are hard to deliver at scale. Constructed-response tasks, such as short answers or explanations, capture students’ reasoning more directly than multiple-choice formats, but they are substantially more time-consuming to assign and evaluate. Furthermore, they can exhibit nontrivial scorer disagreement even under rubric-based evaluation [19] . Notably, this disagreement often stems from the linguistic diversity and variability inherent in st","cbCaikY8URFdxwWw","https://ap.wps.com/l/cbCaikY8URFdxwWw","pdf",944258,12,"English","# Abstract\n# 1. Introduction","[{\"question\":\"Why are accurate guidelines important for LLM-based automated grading?\",\"answer\":\"LLM graders rely on prompt instructions and rubric operationalization to make decisions. If guidelines are ambiguous or poorly specified, the model can misinterpret expert intent and produce unreliable scores.\"},{\"question\":\"What problem does “rule dilution” create in existing rubric optimization approaches?\",\"answer\":\"When frameworks aggregate independent and unstructured error samples into a single update step, conflicting constraints can weaken the model’s grading logic, reducing accuracy.\"},{\"question\":\"How does CARO improve rubric optimization accuracy and efficiency?\",\"answer\":\"CARO decomposes monolithic error signals into distinct confusion-matrix modes, diagnoses and repairs specific misclassification patterns individually, and synthesizes targeted fixing patches selected with a diversity-aware mechanism to avoid guidance conflicts.\"}]","Confusion-Aware Rubric Optimization for LLM-based Automated Grading | PDF",1792054571]