Detecting Answer-Driven Reasoning in LLM-Based Educational Tutors via Truncated Chain-of-Thought Auditing

Jul 16, 2026

Upload by Maeve

Large language model (LLM) tutors can generate fluent step-by-step explanations, yet correctness of the final response does not ensure the reasoning truly originates from the student-facing problem. In tutoring deployments, models may also access private teacher notes such as answer keys or rubrics. This work tests whether such private answer information makes explanations answer-driven by forcing early answers and auditing via Truncated Reasoning AUC Evaluation (TRACE). Results on GSM8K show answer-key access sharply increases early gold recoverability.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages5

Reading Time13 min

Introduction

Method

Detecting Answer-Driven Reasoning in LLM-Based Educational Tutors via Truncated Chain-of-Thought Auditing Bonan Shen Independent Researcher United States [shenbonan2@gmail.com](shenbonan2@gmail.com) Dingyan Shang Independent Researcher United States [dingyanshang@gmail.com](dingyanshang@gmail.com) Youting Wang Northeastern University Boston, MA, USA [ginkoin613@gmail.com](ginkoin613@gmail.com) Tao Ning Syracuse University Syracuse, NY, USA[ntgd1102@gmail.com](ntgd1102@gmail.com) arXiv :2607 .04572v 1 [ cs .AI] 6 Jul 2026 Abstract—Large language model (LLM) tutors often produce fluent step-by-step explanations, but a correct and pedagogically formatted response does not guarantee that the answer was derived from the student-facing problem. In realistic tutoring systems, the model may also have access to teacher notes, answer keys, rubrics, or retrieved solution artifacts. We study whether such private answer information can make tutor explanations answer-driven: the final answer is behaviorally available before the written explanation has justified it. Using Truncated Reasoning AUC Evaluation (TRACE), which probes how early a chain-of-thought prefix can pass a verifier, we evaluate 1000 GSM8K test problems under three paired tutoring contexts: question-only, correct answer-key, and wrong answer-key. At fixed fractions of each generated explanation, we force the model to answer immediately and verify the response against the gold numeric answer. With Qwen2.5-3B-Instruct, answer-key access raises median TRACE AUC from 0.375 to 0.900 and makes the gold answer available at the first 10% prefix in 997 of 1000 cases. The effect remains strong on the 746 examples where both question-only and answer-key explanations end with the correct answer. These results support truncated CoT auditing asa lightweight process-level diagnostic for answer-driven reasoning in math tutoring explanations. Index Terms—AI tutoring, chain-of-thought auditing, answer leakage, educational AI, large language models I. INTRODUCTION LLM tutors are increasingly asked to provide step-by-step help rather than only final answers. This pedagogical format creates a subtle reliability problem: a response can look like a derivation even when it is anchored by private answer-key information. Existing answer-leakage evaluations mainly ask whether the tutor reveals the solution to the learner. We ask a process-level counterpart: does the tutor’s explanation rely on answer information before the written reasoning has earned it? Consider a GSM8K word problem about sunscreen: Pamela uses one ounce per outside hour, spends 4 hours outside per day for 8 days, and has 8-ounce bottles. In the question-only condition, the model eventually gives the correct answer, 4, but forced answers from the first 10%, 25%, 50%, and 75% of its explanation all fail the verifier; the gold answer appears only at the full explanation. When a private teacher-only answer key is added, the visible explanation remains a step-by-step derivation, yet the forced answer is correct from the first 10% TABLE I MOTIVATING EXAMPLES FROM THE GSM8K RUN . EACH CELL REPORTS THE FORCED ANSWER AFTER A COT PREFIX; GREEN CHECKMARKED CELLS MATCH THE GOLD ANSWER , AND RED CELLS FAIL . ABRIDGED PROBLEM STATEMENTS ARE SHOWN FOR SPACE . | Context Final AUC Forced answer at CoT prefix 10% 25% 50% 75% 100% | | --- | | Sunscreen, gold = 4 . Abridged: Pamela uses one ounce of sunscreen per outside hour. She will be outside 4 hours per day for 8 days, and bottles contain 8 ounces. Question-only Answer-key Wrong-key 44 5 0.125 0.900 0.000 – 8 2 5 4 ✓ 4 ✓ 4 ✓ 4 ✓ 4 ✓ 4 ✓ 5 5 5 5 5 | | Flower soil, gold = 3 . Abridged: Artemis has 30 pounds of soil. Sunflowers need 3 pounds each, carnations need 1.5 pounds each, and roses need 1 pound each. They plant 4 sunflowers and 10 carnations. Question-only Answer-key Wrong-key 33 4 0.125 0.900 0.250 10 12 11 9 3

FAQs

What effect does providing an answer key have on the tutoring explanations?

With Qwen2.5-3B-Instruct, answer-key access raises median TRACE AUC from 0.375 to 0.900 and makes the gold answer available at the first 10% prefix in 997 out of 1000 cases. This indicates the explanation can become answer-driven very early when private answer information is available.

Jul 16, 2026

Upload by Maeve