[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-seo-585386-105":3,"detail-sidebar-cat-0-en-105":81,"doc-detail-585386-en":130},{"code":4,"msg":5,"data":6},0,"ok",{"site_id":7,"language":8,"slug":9,"title":10,"keywords":11,"description":12,"schema_data":13,"social_meta":74,"head_meta":76,"extra_data":78,"updated_unix":80},105,"en","a-psychometric-framework-for-context-conditional-evaluation-of-llm-tutoring-in-programming-education-paper","A Psychometric Framework for Context-Conditional Evaluation of LLM Tutoring in Programming Education - Paper","","LLM-based chatbots are increasingly used as on-demand tutors in computer science, but evaluating pedagogical effectiveness is difficult because rubric scoring is costly and aggregated scores mix response quality with rater severity, criterion difficulty, and student-query context. The study proposes a Bayesian explanatory many-facet Rasch model placing rubric-criterion endorsements on a common logit scale, decomposing ratings into latent response quality, criterion difficulty, and rater severity while using query-level covariates for criterion-specific effects.",{"@graph":14,"@context":73},[15,34,56],{"@type":16,"itemListElement":17},"BreadcrumbList",[18,23,27,31],{"item":19,"name":20,"@type":21,"position":22},"https://docshare.wps.com","Home","ListItem",1,{"item":24,"name":25,"@type":21,"position":26},"https://docshare.wps.com/document/","Document",2,{"item":28,"name":29,"@type":21,"position":30},"https://docshare.wps.com/document/research-report/","Research & Report",3,{"item":32,"name":10,"@type":21,"position":33},"https://docshare.wps.com/document/a-psychometric-framework-for-context-conditional-evaluation-of-llm-tutoring-in-programming-education-paper/585386/",4,{"url":32,"name":10,"@type":35,"image":36,"author":41,"headline":10,"publisher":44,"fileFormat":47,"inLanguage":8,"description":12,"dateModified":48,"datePublished":49,"encodingFormat":47,"isAccessibleForFree":50,"interactionStatistic":51},"DigitalDocument",{"url":37,"@type":38,"width":39,"height":40},"https://docshare.wps.com/thumbnails/a-psychometric-framework-for-context-conditional-evaluation-of-llm-tutoring-in-programming-education-paper/585386.png","ImageObject",300,407,{"name":42,"@type":43},"Aria Callaghan","Person",{"url":19,"name":45,"@type":46},"DocShare","Organization","application/pdf","2026-11-05","2026-10-15",true,{"@type":52,"interactionType":53,"userInteractionCount":55},"InteractionCounter",{"@type":54},"ViewAction",7,{"@type":57,"mainEntity":58},"FAQPage",[59,65,69],{"name":60,"@type":61,"acceptedAnswer":62},"Why is evaluating LLM tutoring effectiveness difficult with rubric scoring?","Question",{"text":63,"@type":64},"Rubric scoring is costly, and raw aggregates confound response quality with rater severity, criterion difficulty, and the student-query context.","Answer",{"name":66,"@type":61,"acceptedAnswer":67},"What model does the paper propose for analyzing rubric ratings?",{"text":68,"@type":64},"It proposes a Bayesian explanatory many-facet Rasch model that maps rubric-criterion endorsements onto a common logit scale and decomposes ratings into latent response quality, criterion difficulty, and rater severity.",{"name":70,"@type":61,"acceptedAnswer":71},"What do the results suggest about which rubric criteria are hardest to satisfy?",{"text":72,"@type":64},"The diagnosis criteria are most difficult to meet, while accuracy and next-step criteria are the easiest.","https://schema.org",{"og:url":32,"og:type":75,"og:title":10,"og:site_name":45,"og:description":12},"article",{"robots":77,"canonical":32},"index,follow",{"doc_id":79,"site_id":7},585386,1792193183,{"code":4,"msg":82,"data":83},"success",[84,88,92,96,101,106,110,114,119,122,126],{"id":22,"doc_module":4,"doc_module_name":25,"category_name":85,"show_sort_weight":86,"slug":87},"Story & Novel",90,"story-novel",{"id":26,"doc_module":4,"doc_module_name":25,"category_name":89,"show_sort_weight":90,"slug":91},"Literature",80,"literature",{"id":33,"doc_module":4,"doc_module_name":25,"category_name":93,"show_sort_weight":94,"slug":95},"Exam",70,"exam",{"id":97,"doc_module":4,"doc_module_name":25,"category_name":98,"show_sort_weight":99,"slug":100},5,"Comic",60,"comic",{"id":102,"doc_module":4,"doc_module_name":25,"category_name":103,"show_sort_weight":104,"slug":105},6,"Technology",50,"technology",{"id":55,"doc_module":4,"doc_module_name":25,"category_name":107,"show_sort_weight":108,"slug":109},"Healthcare",40,"healthcare",{"id":111,"doc_module":4,"doc_module_name":25,"category_name":29,"show_sort_weight":112,"slug":113},8,30,"research-report",{"id":115,"doc_module":4,"doc_module_name":25,"category_name":116,"show_sort_weight":117,"slug":118},9,"Religion & Spirituality",20,"religion-spirituality",{"id":117,"doc_module":4,"doc_module_name":25,"category_name":120,"show_sort_weight":117,"slug":121},"World Cup","world-cup",{"id":123,"doc_module":4,"doc_module_name":25,"category_name":124,"show_sort_weight":123,"slug":125},10,"Lifestyle","lifestyle",{"id":127,"doc_module":4,"doc_module_name":25,"category_name":128,"show_sort_weight":97,"slug":129},19,"General","general",{"code":4,"msg":82,"data":131},{"doc_id":79,"user_id":132,"nickname":42,"user_avatar":133,"doc_module":4,"category_id":111,"category_name":29,"doc_title":10,"doc_description":12,"doc_content":134,"file_id":135,"file_url":136,"file_type":137,"file_size":138,"view_count":55,"is_deleted":4,"is_public":22,"is_downloadable":22,"audit_status":22,"page_count":139,"language":140,"language_code":8,"site_id":7,"html_lang":8,"table_of_contents":141,"faqs":142,"seo_title":143,"seo_description":12,"update_tm":144,"read_time":145,"preview_page_count":4},962084926284,"https://ap-avatar.wpscdn.com/davatar_29158cc5080c5b710cf443261637dec0","A Psychometric Framework for Context-Conditional Evaluation of LLM Tutoring in Programming Education  \nJinnie Shin  \nUniversity of Florida  \n[jinnie.shin@ufl.edu](jinnie.shin@ufl.edu)  \nZhe Li  \nUniversity of Florida  \n[zheli@ufl.edu](zheli@ufl.edu)  \nPauline Aguinalde  \nUniversity of Florida  \n[aaguinalde@ufl.edu](aaguinalde@ufl.edu)  \nLaura M. Cruz Castro University of Florida  \n[cruzcastrol@ufl.edu](cruzcastrol@ufl.edu)  \nABSTRACT  \nLLM-based chatbots are increasingly used as on-demand tutors in computer science education, yet evaluating pedagogical effectiveness remains difficult because rubric scoring is costly and raw aggregates confound response quality with rater severity, criterion difficulty, and student-query context. We propose a Bayesian explanatory many-facet Rasch model (MFRM) that places rubric-criterion endorsementson a common logit scale and decomposes ratings into latent response quality, criterion difficulty, and rater severity, while incorporating query-level covariates as criterion-specific effects. Applied to 249 CodeHelp exchanges from a large undergraduate C++ course, double-coded by two experts on mistake identification, mistake location, guidance, accuracy of guidance, and correctness of next steps, the model indicates minimal systematic rater differences and pronounced criterion-difficulty gaps. Diagnosis criteria are the most difficult to satisfy, whereas accuracy and next-step criteria are the easiest. Input sufficiency yields modest, criteriondependent shifts in endorsement probability, and topic patterns are explored descriptively, underscoring that observed tutoring quality depends on what students provide and the type of help they request.  \nKeywords  \neducational chatbots; large language models; pedagogical effectiveness; Many-facet Rasch Model  \n1. INTRODUCTION  \nLLM-based chatbots, also referred to as conversational agents, are increasingly deployed as on-demand tutors across K–12 and higher education in a wide range of domains [28, 33] . In computer science education in particular, a growing body of work reports benefits for programming-focused learning tasks, including answering students’ questions [36], supporting learning of core programming concepts [32], and assisting with assessment or performance feedback [12, 49] . Unlike earlier chatbots that relied on scripted rules or narrow task-specific models, LLM-based chatbots are generative, Jinnie Shin, Zhe Li, Pauline Aguinalde, and Laura Cruz Castro. APsychometric Framework for Context-Conditional Evaluation of LLM Tutoring in Programming Education. In Anthony Botelho, Maria Mercedes T. Rodrigo, Adish Singla, Hiroaki Ogata, Hyojeong So, and Young Hoan Cho (eds.) Proceedings of the 19th International Conference on Educational Data Mining, Seoul, Republic of Korea, June, 2026, pp. 29–43. International Educational Data Mining Society (2026) .  \n© 2026 Copyright is held by the author(s) . This work is distributed under the Creative Commons Attribution NonCommercial NoDerivatives 4.0 International (CC BY-NC-ND 4.0) license. [https://doi.org/10.5281/zenodo.21039780](https://doi.org/10.5281/zenodo.21039780)  \nwith an enhanced capacity to produce open-ended, contextsensitive explanations and guidance specific to a learner’s prompt. Recent studies have therefore examined a range of LLM-powered tools, including general-purpose models (e.g. , GPTs) and code-specialized assistants (e.g., Codex-and Copilot style systems), with emerging evidence that these tools can reshape both pedagogical interactions (e.g., debugging support, conceptual explanation) and assessment-related practices (e.g., feedback generation and evaluation) [55, 19] .  \nAt the same time, rapid adoption has intensified the need for rigorous evaluation frameworks that can determine not only whether these systems work, but also how, for whom, and under what conditions they support learning and meaningful educational interactions [21, 1, 28] . To date, evaluation evidence is often ass","cbCaiu3S2VfeuJkS","https://ap.wps.com/l/cbCaiu3S2VfeuJkS","pdf",1325120,15,"English","# Abstract\n# 1. Introduction\n## Background on LLM tutoring and evaluation needs\n## Limitations of existing metrics\n## Rasch-family approaches to LLM evaluation","[{\"question\":\"Why is evaluating LLM tutoring effectiveness difficult with rubric scoring?\",\"answer\":\"Rubric scoring is costly, and raw aggregates confound response quality with rater severity, criterion difficulty, and the student-query context.\"},{\"question\":\"What model does the paper propose for analyzing rubric ratings?\",\"answer\":\"It proposes a Bayesian explanatory many-facet Rasch model that maps rubric-criterion endorsements onto a common logit scale and decomposes ratings into latent response quality, criterion difficulty, and rater severity.\"},{\"question\":\"What do the results suggest about which rubric criteria are hardest to satisfy?\",\"answer\":\"The diagnosis criteria are most difficult to meet, while accuracy and next-step criteria are the easiest.\"}]","A Psychometric Framework for Context-Conditional Evaluation of LLM Tutoring in Programming Education - Paper | PDF",1792054578,38]