Reflective Dialogue or Prompt Refinement? Effects of Tutor Scaffolding on Students' Independent LLM Use for Programming

Sep 21, 2026

Uploaded by Liam

Large Language Models (LLMs) can offer personalized learning support, yet educational studies highlight that outcomes depend strongly on how students interact with these tools. This work compares two tutor designs: a Socratic-Guidance (SG) tutor using dialogic questioning and a Prompt-Refinement (PR) tutor guiding effective prompt formulation. A two-phase graduate mobile robotics study with 66 students in a six-week intervention, then 52 using an unconstrained LLM, evaluates prompting practices, learning gains, and later LLM use.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages15

Reading Time38 min

AccessFree to download

Abstract

Introduction

Cognitive and metacognitive risks of LLM use in learning

Improving the effectiveness of LLMs for learning

arXiv :2607 .03303v 1 [ cs .AI] 3 Jul 2026 Reflective Dialogue or Prompt Refinement? Effects of Tutor Scaffolding on Students’ Independent LLM Use for Programming⋆ J´erˆome Brender 1 ,2[0009−0008−3279−5641], Laila El-Hamamsy2[0000−0002−6046−4822], Kim Uittenhove 1[0000−0001−5450−3875], Aitor Perez3[0000−0001−6412−3235], Patrick Jermann3[0000−0001−9199−2831], Francesco Mondada 1[0000−0001−8641−8704], and Engin Bumbacher2[0000−0002−4322−7059] 1 MOBOTS & LEARN, EPFL, Switzerland [jerome.brender@epfl.ch](jerome.brender@epfl.ch) , [kim.uittenhove@epfl.ch](kim.uittenhove@epfl.ch) , [francesco.mondada@epfl.ch](francesco.mondada@epfl.ch) 2 University of Teacher Education (Haute ´Ecole P´edagogique) Vaud, Switzerland engin.bumbacher@hepl .ch, laila .elhamamsy@hepl .ch 3 Center for Digital Education & LEARN, EPFL, Switzerland [aitor.perez@epfl.ch](aitor.perez@epfl.ch) , [patrick.jermann@epfl.ch](patrick.jermann@epfl.ch) Abstract. While Large Language Models (LLMs) can provide personalized support in learning, several studies have raised concerns regarding their use in education. Importantly, learning depends on how students engage with LLMs. This study examined how two types of LLM-based tutors shape students’ prompting practices, learning, and subsequent LLM-use: a Socratic-Guidance (SG) tutor, which structures interaction through dialogic questioning, and a Prompt-Refinement (PR) tutor that guides the formulation of effective prompts. We conducted a two-phase study in a graduate-level mobile robotics course: 66 students used either the SG or PR tutor during a 6-week intervention, followed by 52 students using an unconstrained LLM during a 3-week course project. Results show that while the SG-and PR tutors led to similar task performance and prompting patterns during guided use, they differ in learning outcomes and later LLM-use. SG-students, relative to PR-student, achieved higher learning gains in later sessions, and were more likely to adopt understanding-driven prompting strategies, which are predictive of higher understanding, when using an unconstrained LLM. Although learners perceived the SG tutor as less efficient, the findings suggest that Socratic guidance supports the development of students’ capacity to learn with LLMs over time, highlighting its importance for LLM tutor design. Keywords: Large Language Models · Computing Education · Behaviors · Performance · Learning · Effectiveness · Tutor ⋆ We thank our colleagues (M.B, A.B, L.C, E.C, M.D, A.S, G.D, V.D, F.E) for their support, and the students who generously volunteered to participate in the study. 2 J. Brender et al. 1 Introduction With the rise of Large Language Models (LLMs), a growing body of work has examined applications of LLM-based tools in education, including curriculum development, automated assessment and feedback, and tools that provide personalized support during learning activities [15,25] . At the same time, researchers have argued that educational benefits are not inherent to such LLM-based systems but depend on their pedagogical design and integration, and have warned of substantial risks when these conditions are not met [15] . 1.1 Cognitive and metacognitive risks of LLM use in learning One central risk associated with students’ use of LLM-based systems is the conflation of improved task performance with learning. While studies have found that students who use such systems to solve problems often achieve higher performance, these gains are neither indicative of students’ ability to perform without the tool4 nor of improved learning outcomes5 . This disconnect between performance and learning can be partly explained through research on cognition and metacognition. Several studies have raised concerns that students’ use of LLMs may foster metacognitive laziness [7,10,24] . As LLMs become more capable, students may increasingly rely on these systems in ways that reduce their active cognitive and metacognitive engagement

FAQs

What tutor types does the study compare?

It compares a Socratic-Guidance (SG) tutor that structures interaction via dialogic questioning with a Prompt-Refinement (PR) tutor that guides students to formulate effective prompts.

How was the study conducted and in what learning context?

The authors run a two-phase study in a graduate-level mobile robotics course: 66 students use either SG or PR during a six-week intervention, followed by 52 students using an unconstrained LLM during a three-week course project.

What learning and later LLM-use differences were found between SG and PR?

SG and PR led to similar task performance and prompting patterns during guided use, but SG students showed higher later learning gains and were more likely to adopt understanding-driven prompting strategies when using an unconstrained LLM.

Sep 21, 2026

Uploaded by Liam