[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-85280-en":3,"doc-seo-85280-105":29,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},85280,1374391974585,"Genevieve","https://ap-avatar.wpscdn.com/davatar_276721f389ce27ea32af1340a28f341c",8,"Research & Report","LLM-Generated Design Problems for Assessing Higher-Order Thinking in Project-Based Learning","Project-based learning (PjBL) is widely used in computing education, yet traditional assessments often miss higher-order thinking (HOT), particularly when learners must transfer skills to new situations. This study proposes “design problems” (DPs): short, scenario-based prompts that require applying project concepts in unfamiliar contexts. Instructor surveys, LLM-based DP generation evaluation, and student performance data show instructors value DPs, while LLMs reduce authoring effort and yield strong expert agreement. Results suggest DPs assess HOT aspects distinct from traditional project grades, with keystroke evidence indicating deeper planning and revision.","LLM-Generated Design Problems for Assessing Higher-Order Thinking in Project-Based Learning  \nAhmad D. Suleiman  \n[as4300@rit.edu](as4300@rit.edu)  \nRochester Institute of Technology Rochester, NY, USA  \nDaqing Hou  \n[dqvse@rit.edu](dqvse@rit.edu)[ ](dqvse@rit.edu)Rochester Institute of Technology  \nRochester, NY, USA  \nMaliha Noushin Raida  \n[mr6074@rit.edu](mr6074@rit.edu)  \nRochester Institute of Technology Rochester, NY, USA  \narXiv :2607 . 11032v1 [ cs .CY] 13 Jul 2026  \nAbstract  \nProject-based learning (PjBL) is common in computing education, but traditional assessments ofPjBL often fail to capture higher-order thinking (HOT), especially in transfer contexts. This study introduces “design problems”(DPs): concise, scenario-based prompts that require applying project concepts in new situations, to address this gap. We examined instructor perceptions, the ability of large language models (LLMs) to generate DPs, and student experiences. Surveys of 31 instructors, evaluation of 80 LLM-generated DPs, and student performance data showed that while instructors value DPs, creation effort is a barrier. LLMs helped by producing high-quality prompts with strong expert agreement. Students rated DPs from different LLMs similarly, and their performance on DP tasks showed negligible correlation with traditional project grades, suggesting DPs may capture distinct aspects of HOT. Keystroke data also suggested deeper cognitive engagement of students through planning and revision behaviors. Overall, DPs appear to be a useful complement to traditional assessments, especially in situations where AI use or collaboration may undermine individual learning.  \nCCS Concepts  \n• Social and professional topics → Computing education; Student assessment; • Computing methodologies → Artificial intelligence.  \nKeywords  \nProject-based Learning, Higher-order Thinking, Design Problems, Assessment, Large Language Models  \nACM Reference Format:  \nAhmad D. Suleiman, Daqing Hou, and Maliha Noushin Raida. 2026. LLMGenerated Design Problems for Assessing Higher-Order Thinking in ProjectBased Learning. In Proceedings of the 2nd ACM Virtual Global Computing Education Conference V.1 (SIGCSE Virtual 2026), November 12–15, 2026, Virtual Event, USA. ACM, New York, NY, USA, 7 pages. [https://doi.org/10.1145/](https://doi.org/10.1145/)[ ](https://doi.org/10.1145/)3795867.3831014  \n1 Introduction  \nProject-based Learning (PjBL) is widely adopted in computing education due to its effectiveness in fostering practical skills and deeper cognitive engagement [31] . Traditional PjBL assessment practices are mainly based on project artifacts, such as code, reports, and  \nThis work is licensed under a Creative Commons Attribution 4 .0 International License. SIGCSE Virtual 2026, Virtual Event, USA  \n© 2026 Copyright held by the owner/author(s) .  \nACM ISBN 979-8-4007-2506-7/2026/11  \n[https://doi.org/10.1145/3795867.3831014](https://doi.org/10.1145/3795867.3831014)  \nScenario:  \nA healthcare technology company wants to create an application to help fam lies and caregivers mon tor the well being of elderly individuals One early indicator of certain health issues is a change in fine motor skills, which refers to the small, precise movements of the hands and fingers. The company believes that how a person interacts with their smartphone’s touchscreen could reveal subtle, long-term changes in these skills.  \nProblem:  \nPropose a high level design for a system that uses a person’s daily smartphone usage to detect potential declines in their motor skills. Your description should outline the major components of your system and explain how information would flow between them to achieve the goal. Specifically, describe (1) how you would capture relevant user interaction data from the phone,(2) how this information would be stored and managed centrally, and (3) how you would process this information to establish a personal baseline and identify signif cant, long term deviations from it.  ","cbCaiaokArOjNbP8","https://ap.wps.com/l/cbCaiaokArOjNbP8","pdf",570948,1,7,"English","en",105,"# Introduction\n## Project-based learning assessment challenges\n## Design problems as a complementary assessment approach\n## Transfer learning and higher-order thinking focus","[{\"question\":\"什么是“设计问题”(Design Problems, DPs)？\",\"answer\":\"DPs是简短的、情境化的提示任务，要求学生在新情境中应用项目所学概念，从而触发分析、评价与创造等高阶思维活动。它与原项目学习目标对齐，但场景与呈现方式保持区分性。\"},{\"question\":\"研究如何检验DPs是否能评估更高阶思维？\",\"answer\":\"研究通过三类数据进行考察：31位授课者的看法、对80个由LLM生成的DP的评估，以及学生在DP任务中的表现数据与传统项目成绩的对照。结果显示，DP任务表现与传统项目成绩几乎不相关，提示DPs可能捕捉到不同的HOT维度。\"},{\"question\":\"LLM在DP生成中起到怎样的作用？\",\"answer\":\"LLM被用于生成DP，并在质量与专家一致性方面表现良好，能够产出高质量提示，从而降低教师撰写DP的投入。研究也指出教师创建DP的努力成本仍是现实障碍。\"}]",1784202230,18,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":27},"llm-generated-design-problems-for-assessing-higher-order-thinking-in-project-based-learning","",{"@graph":35,"@context":85},[36,53,68],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/llm-generated-design-problems-for-assessing-higher-order-thinking-in-project-based-learning/85280/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-17","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"什么是“设计问题”(Design Problems, DPs)？","Question",{"text":75,"@type":76},"DPs是简短的、情境化的提示任务，要求学生在新情境中应用项目所学概念，从而触发分析、评价与创造等高阶思维活动。它与原项目学习目标对齐，但场景与呈现方式保持区分性。","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"研究如何检验DPs是否能评估更高阶思维？",{"text":80,"@type":76},"研究通过三类数据进行考察：31位授课者的看法、对80个由LLM生成的DP的评估，以及学生在DP任务中的表现数据与传统项目成绩的对照。结果显示，DP任务表现与传统项目成绩几乎不相关，提示DPs可能捕捉到不同的HOT维度。",{"name":82,"@type":73,"acceptedAnswer":83},"LLM在DP生成中起到怎样的作用？",{"text":84,"@type":76},"LLM被用于生成DP，并在质量与专家一致性方面表现良好，能够产出高质量提示，从而降低教师撰写DP的投入。研究也指出教师创建DP的努力成本仍是现实障碍。","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,119,122,127,130,134],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":45,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":45,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":21,"doc_module":4,"doc_module_name":45,"category_name":116,"show_sort_weight":117,"slug":118},"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":120,"slug":121},30,"research-report",{"id":123,"doc_module":4,"doc_module_name":45,"category_name":124,"show_sort_weight":125,"slug":126},9,"Religion & Spirituality",20,"religion-spirituality",{"id":125,"doc_module":4,"doc_module_name":45,"category_name":128,"show_sort_weight":125,"slug":129},"World Cup","world-cup",{"id":131,"doc_module":4,"doc_module_name":45,"category_name":132,"show_sort_weight":131,"slug":133},10,"Lifestyle","lifestyle",{"id":135,"doc_module":4,"doc_module_name":45,"category_name":136,"show_sort_weight":106,"slug":137},19,"General","general"]