Tracking Performance Gains in AI-Supported Learning - A Longitudinal Study of Criterion-Referenced AI Feedback on StudyPulse

Sep 20, 2026

Uploaded by Dipper

A longitudinal analysis evaluates 28,279 graded StudyPulse submissions from February–November 2025, using authentic Victorian Curriculum and Assessment Authority (VCAA) past examination questions across 2018–2024. Performance is difficulty-adjusted via z-score normalization against official VCAA score distributions representing 50,000+ students per year. StudyPulse users show a mean advantage of ~0.7–0.8 standard deviations by the study end, roughly +5 VCE study score points and +10 ATAR points, with mean improvement of about +0.55 from early to late-year performance. A normal-distribution model estimates ~3× higher odds of attaining a 40+ study score (~23% vs ~8%)

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages16

Reading Time40 min

AccessFree to download

Introduction

Background

Why Authentic VCAA Questions Matter

Why Immediate Feedback Matters

Research Questions

Data and Methodology

Data Collection

StudyPulse Research Team Tracking Performance Gains in AI-Supported Learning: A Longitudinal Study of Criterion-Referenced AI Feedback on StudyPulse StudyPulse Research Team November 2025 Abstract We present a longitudinal analysis of 28,279 graded submissions from StudyPulse users (February–November 2025), all on authentic Victorian Curriculum and Assessment Authority (VCAA) past examination questions spanning 7 years of real VCE exams (2018–2024) . Performance is difficulty-adjusted using z-score normalisation against official VCAA score distributions, which represent cohorts of 50,000+ students per year. By the end of the study window, StudyPulse users achieve a mean performance level of approximately 0 .7–0.8 standard deviations above the exam population average, corresponding to roughly +5 VCE study score points (from 30 to ≈ 35) and +10 ATAR points (from 70 to ≈ 80) . Within the observation period, we observe a mean improvement of about +0 .55 standard deviations from early-to lateyear performance. Under a normal-distribution model, we further estimate that StudyPulse users are approximately 3 × more likely to achieve a 40+ study score compared to the general student population (≈ 23% vs. ≈ 8%) . Keywords: Educational Technology, AI Tutoring, Z-Score Analysis, VCE, Learning Analytics 1 Introduction 1.1 Background The Victorian Certificate of Education (VCE) uses a standardised scoring system in which study scores are scaled to a population mean of 30 and standard deviation of 7 . This normalisation allows for meaningful comparison across subjects and years. Similarly, the Australian Tertiary Admission Rank (ATAR) is typically modelled as an approximately normal distribution for the ATAR-eligible cohort, with mean around 70 and standard deviation around 14 . In this work we exploit these scaling conventions to place StudyPulse users’ performance on a difficulty-adjusted scale that is directly comparable to population-level exam performance. 1.2 Why Authentic VCAA Questions Matter This study is based exclusively on real VCAA past examination questions from the last 7 years of VCE exams (2018–2024) . This choice is critical for three reasons: 1. Validity. Students are practicing on the exact question formats, rubrics, and difficulty levels they will encounter in actual exams. StudyPulse Research Team 2. Reliable benchmarks. VCAA publishes official score distributions for each question, representing 50,000+ real student responses per year, which provide a robust population benchmark. 3. No synthetic inflation. Unlike practice questions or third-party materials, VCAA questions have known and fixed difficulty profiles that cannot be artificially adjusted to make gains appear larger than they are. 1.3 Why Immediate Feedback Matters The value of instant, criterion-referenced feedback is well-established in learning science. Hattie and Timperley’s synthesis of over 800 meta-analyses identifies feedback as one of the highest-impact influences on achievement, with an effect size of d ≈ 0.7, nearly double the average effect of a year of schooling [4] . Crucially, feedback must be timely: when corrections arrive days or weeks after the student has forgotten their reasoning, the opportunity to reshape underlying skills has largely passed. Neuroimaging research supports this: immediate feedback engages striatal systems associated with procedural learning and skill automaticity, whereas delayed feedback shifts learning toward declarative memory systems better suited to memorising facts than tuning exam technique [5] . Recent data quantifies the behavioural difference. In a study of approximately 19,500 Australian and New Zealand secondary students, Education Perfect found that when AI feedback was immediate, 83% of students with a low-quality response attempted a revision; without immediate feedback, only 7% did [6] . Instant feedback transforms assessment from a one-shot judgment into an iter

FAQs

What dataset and time window does the longitudinal study use?

The study analyzes 28,279 graded submissions from StudyPulse users submitted between February and November 2025.

How is performance adjusted to allow comparisons across question difficulty?

Performance is difficulty-adjusted using z-score normalization against official VCAA score distributions.

What improvement and outcome probabilities are estimated for StudyPulse users?

By the end of the observation window, users average about 0.7–0.8 standard deviations above the exam population average, and under a normal-distribution model they are estimated to be about 3× more likely to achieve a 40+ study score (~23% vs ~8%).

Sep 20, 2026

Uploaded by Dipper