From Human Days to Machine Seconds: Automatically Answering and Generating Machine Learning Final Exams

Aug 3, 2026

Upload by Himbo

A machine learning final exam at leading universities typically requires faculty days to author and students hours to solve. Large language models are shown to pass machine learning finals at a human level on publicly available exam questions after training, and to automatically generate new, high-quality exam questions within seconds. A curated online dataset and benchmark enable answering and question generation, supported by automated checkers for multiple-choice, numeric, and expression answers, with ablation studies comparing prompting and few-shot strategies.

LanguageEnglish

CategoryTechnology

FormatPDF

Pages9

Reading Time23 min

AccessFree to download

Introduction

Dataset and benchmark

Automated answering and checkers

Few-shot vs zero-shot and prompting ablations

Meta-question evaluation

From Human Days to Machine Seconds: Automatically Answering and Generating Machine Learning Final Exams arXiv :2206 .05442v7 [ cs .LG] 28 Jun 2023 Iddo Drori MIT, Columbia University, BU Cambridge, USA [idrori@mit.edu](idrori@mit.edu) Sarah Zhang MIT Cambridge, USA [sazhang@mit.edu](sazhang@mit.edu) Pedro Lantigua MIT Cambridge, USA[lantigua@mit.edu](lantigua@mit.edu) Sarah J. Zhang MIT Cambridge, USA [sjzhang@mit.edu](sjzhang@mit.edu) Keith Tyser Boston University Boston, USA [ktyser@bu.edu](ktyser@bu.edu) Saisamrit Surbehera Columbia University New York, USA [ss6365@columbia.edu](ss6365@columbia.edu) Reece Shuttleworth MIT Cambridge, USA[rshuttle@mit.edu](rshuttle@mit.edu) Zad Chin Harvard University Cambridge, USA[zadchin@college.harvard.edu](zadchin@college.harvard.edu) Gregory Hunter Columbia University New York, USA[geh2129@columbia.edu](geh2129@columbia.edu) Derek Austin Columbia University New York, USA[da2986@columbia.edu](da2986@columbia.edu) Leonard Tang Harvard University Cambridge, USA [leonardtang@college.harvard.edu](leonardtang@college.harvard.edu) Yann Hicke Cornell University Ithaca, USA[ylh8@cornell.edu](ylh8@cornell.edu) Sage Simhon MIT Cambridge, USA [simhon@mit.edu](simhon@mit.edu) Sathwik Karnik MIT Cambridge, USA [skarnik@mit.edu](skarnik@mit.edu) Darnell Granberry MIT Cambridge, USA [darnellg@mit.edu](darnellg@mit.edu) Madeleine Udell Stanford University Stanford, USA [udell@stanford.edu](udell@stanford.edu) ABSTRACT A ﬁnal exam in machine learning at a top institution such as MIT, Harvard, or Cornell typically takes faculty days to write, and students hours to solve. We demonstrate that large language models pass machine learning ﬁnals at a human level, on ﬁnals available online after the models were trained, and automatically generate new human-quality ﬁnal exam questions in seconds. Previous work has developed program synthesis and few-shot learning methods to solve university-level problem set questions in mathematics and STEM courses. In this work, we develop and compare methods that solve ﬁnal exams, which diﬀer from problem sets Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for proﬁtor commercial advantage and that copies bear this notice and the full citation on theﬁrst page. Copyrights for third-party components ofthis work must be honored. For all other uses, contact the owner/author(s) . KDD ’23, August 6–10, 2023, Long Beach, CA, USA © 2023 Copyright held by the owner/author(s) . ACM ISBN 979-8-4007-0103-0/23/08 . [https://doi.org/10.1145/3580305.3599827](https://doi.org/10.1145/3580305.3599827) in several ways: the questions are longer, have multiple parts, are more complicated, and span a broader set of topics. We curate adataset and benchmark of questions from machine learning ﬁnal exams available online and code for answering these questions and generating new questions. We show how to generate new questions from other questions and course notes. For reproducibility and future research on this ﬁnal exam benchmark, we use automatic checkers for multiple-choice, numeric, and questions with expression answers. We perform ablation studies comparing zeroshot learning with few-shot learning and chain-of-thought prompting using GPT-3, OPT, Codex, and ChatGPT across machine learning topics and ﬁnd that few-shot learning methods perform best. We highlight the transformative potential of language models to streamline the writing and solution of large-scale assessments, signiﬁcantly reducing the workload from human days to mere machine seconds. Our results suggest that rather than banning large language models such as ChatGPTin class, instructors should teach students to harness them by asking students meta-questions about correctness, completeness, and originality of the responses generated

FAQs

How do the authors make language models answer machine learning final exams?

They curate a structured dataset of online machine learning final exam questions and use automated checkers to evaluate multiple-choice, numeric, and expression-answer formats. The models are prompted to solve and explain, then validated through the benchmark checks.

What is the key result about model performance on final exams?

The best-performing approaches allow large language models to pass machine learning finals at a human level on available exam questions, while also generating new questions quickly.

Which learning strategy performs best in the comparisons, and how is it evaluated?

Few-shot learning methods perform best compared with zero-shot learning and other prompting approaches. The evaluation includes meta-questions assessing correctness, completeness, and originality of the generated responses.

Aug 3, 2026

Upload by Himbo