LLM-Generated Design Problems for Assessing Higher-Order Thinking in Project-Based Learning

Jul 16, 2026

Project-based learning (PjBL) is widely used in computing education, yet traditional assessments often miss higher-order thinking (HOT), particularly when learners must transfer skills to new situations. This study proposes “design problems” (DPs): short, scenario-based prompts that require applying project concepts in unfamiliar contexts. Instructor surveys, LLM-based DP generation evaluation, and student performance data show instructors value DPs, while LLMs reduce authoring effort and yield strong expert agreement. Results suggest DPs assess HOT aspects distinct from traditional project grades, with keystroke evidence indicating deeper planning and revision.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages7

Reading Time18 min

Introduction

Project-based learning assessment challenges

Design problems as a complementary assessment approach

Transfer learning and higher-order thinking focus

LLM-Generated Design Problems for Assessing Higher-Order Thinking in Project-Based Learning Ahmad D. Suleiman [as4300@rit.edu](as4300@rit.edu) Rochester Institute of Technology Rochester, NY, USA Daqing Hou [dqvse@rit.edu](dqvse@rit.edu)[ ](dqvse@rit.edu)Rochester Institute of Technology Rochester, NY, USA Maliha Noushin Raida [mr6074@rit.edu](mr6074@rit.edu) Rochester Institute of Technology Rochester, NY, USA arXiv :2607 . 11032v1 [ cs .CY] 13 Jul 2026 Abstract Project-based learning (PjBL) is common in computing education, but traditional assessments ofPjBL often fail to capture higher-order thinking (HOT), especially in transfer contexts. This study introduces “design problems”(DPs): concise, scenario-based prompts that require applying project concepts in new situations, to address this gap. We examined instructor perceptions, the ability of large language models (LLMs) to generate DPs, and student experiences. Surveys of 31 instructors, evaluation of 80 LLM-generated DPs, and student performance data showed that while instructors value DPs, creation effort is a barrier. LLMs helped by producing high-quality prompts with strong expert agreement. Students rated DPs from different LLMs similarly, and their performance on DP tasks showed negligible correlation with traditional project grades, suggesting DPs may capture distinct aspects of HOT. Keystroke data also suggested deeper cognitive engagement of students through planning and revision behaviors. Overall, DPs appear to be a useful complement to traditional assessments, especially in situations where AI use or collaboration may undermine individual learning. CCS Concepts • Social and professional topics → Computing education; Student assessment; • Computing methodologies → Artificial intelligence. Keywords Project-based Learning, Higher-order Thinking, Design Problems, Assessment, Large Language Models ACM Reference Format: Ahmad D. Suleiman, Daqing Hou, and Maliha Noushin Raida. 2026. LLMGenerated Design Problems for Assessing Higher-Order Thinking in ProjectBased Learning. In Proceedings of the 2nd ACM Virtual Global Computing Education Conference V.1 (SIGCSE Virtual 2026), November 12–15, 2026, Virtual Event, USA. ACM, New York, NY, USA, 7 pages. [https://doi.org/10.1145/](https://doi.org/10.1145/)[ ](https://doi.org/10.1145/)3795867.3831014 1 Introduction Project-based Learning (PjBL) is widely adopted in computing education due to its effectiveness in fostering practical skills and deeper cognitive engagement [31] . Traditional PjBL assessment practices are mainly based on project artifacts, such as code, reports, and This work is licensed under a Creative Commons Attribution 4 .0 International License. SIGCSE Virtual 2026, Virtual Event, USA © 2026 Copyright held by the owner/author(s) . ACM ISBN 979-8-4007-2506-7/2026/11 [https://doi.org/10.1145/3795867.3831014](https://doi.org/10.1145/3795867.3831014) Scenario: A healthcare technology company wants to create an application to help fam lies and caregivers mon tor the well being of elderly individuals One early indicator of certain health issues is a change in fine motor skills, which refers to the small, precise movements of the hands and fingers. The company believes that how a person interacts with their smartphone’s touchscreen could reveal subtle, long-term changes in these skills. Problem: Propose a high level design for a system that uses a person’s daily smartphone usage to detect potential declines in their motor skills. Your description should outline the major components of your system and explain how information would flow between them to achieve the goal. Specifically, describe (1) how you would capture relevant user interaction data from the phone,(2) how this information would be stored and managed centrally, and (3) how you would process this information to establish a personal baseline and identify signif cant, long term deviations from it.

FAQs

什么是“设计问题”(Design Problems, DPs)？

DPs是简短的、情境化的提示任务，要求学生在新情境中应用项目所学概念，从而触发分析、评价与创造等高阶思维活动。它与原项目学习目标对齐，但场景与呈现方式保持区分性。

研究如何检验DPs是否能评估更高阶思维？

研究通过三类数据进行考察：31位授课者的看法、对80个由LLM生成的DP的评估，以及学生在DP任务中的表现数据与传统项目成绩的对照。结果显示，DP任务表现与传统项目成绩几乎不相关，提示DPs可能捕捉到不同的HOT维度。

LLM在DP生成中起到怎样的作用？

LLM被用于生成DP，并在质量与专家一致性方面表现良好，能够产出高质量提示，从而降低教师撰写DP的投入。研究也指出教师创建DP的努力成本仍是现实障碍。

Jul 16, 2026

Upload by Genevieve