Advanced Financial Reasoning at Scale - A Comprehensive Evaluation of Large Language Models on CFA Level III

Sep 4, 2026

As financial institutions increasingly adopt Large Language Models (LLMs), rigorous domain-specific evaluation becomes essential for responsible deployment. This paper benchmarks 23 state-of-the-art LLMs on mock CFA Level III exams covering demanding multiple-choice and essay questions. Models are assessed using zero-shot, chain-of-thought, and self-discover prompting, including both proprietary and open-source systems. Frontier reasoning models show strong performance, with limited separation on multiple-choice tasks but fewer models excelling on complex essays, while cost and latency analysis guides practical selection.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages17

Reading Time43 min

AccessFree to download

Abstract

Keywords

Introduction

Advanced Financial Reasoning at Scale: A Comprehensive Evaluation of Large Language Models on CFA Level III Pranam Shetty 1 , Abhisek Upadhayaya3 , Parth Mitesh Shah2 , Shilpi Nayak2 , Anna Joo Fee2 , Srikanth Jagabathula3 , 1Rochester Institute of Technology, 2 GoodFin, Inc, 3New York University ps9960@rit.edu, {parth, shilpi, anna}@goodfin.com, au2216@nyu.edu, sjagabat@stern.nyu.edu arXiv :2507 .02954v2 [ cs .CL] 22 Sep 2025 Abstract As financial institutions increasingly adopt Large Language Models (LLMs), rigorous domainspecific evaluation becomes critical for responsible deployment. For advanced financial reasoning, the Chartered Financial Analyst (CFA) Level III exam is widely considered the gold standard. In this paper, we present a comprehensive benchmark evaluating 23 state-of-the-art LLMs on mock CFA Level III exams, which require answering challenging multiple choice and essay questions. We evaluate reasoning and non-reasoning models, both proprietary and open source, using three prompting strategies: zero-shot, chain-of-thought, and selfdiscover. We find that frontier reasoning models, such as o4-mini, Gemini 2.5 Pro, and Claude Opus 4, using chain-of-thought prompting demonstrate strong capabilities, successfully passing the mock Level III exams. While there is little to separate the frontier models on multiple choice questions, only a few models excel at the complex essay questions, which require analysis, synthesis, and strategic thinking. These results demonstrate significant progress in the financial reasoning capabilities of LLMs, which previously [13] could clear Level I and Level II exams but struggled with the Level III exam, particularly the essay questions. We also benchmark the models on cost and latency, providing crucial guidance for practitioners on model selection. Our analysis highlights ongoing challenges in cost-effective deployment and underscores the need for nuanced interpretation of performance against professional benchmarks. 1 Keywords: Large Language Models, Financial Reasoning, CFA Level III Benchmark, Chain-of-Thought Prompting, SelfConsistency Prompting, Self-Discover Prompting 1 Introduction Large Language Models (LLMs) are rapidly transforming the financial services industry, with growing adoption for critical 1Accepted at FinLLM@IJCAI 2025 applications including research, education, trading, and client advisory services [7, 8] . As the use of such models proliferates, it is important to ensure that they meet the strict standards necessary for application in high-stakes settings. While the LLMs are demonstrating improving proficiency on general reasoning tasks, a critical question arises: can they also handle specialized, high-stakes analytical reasoning required for professional financial decision-making? [2, 3] To asses the capabilities of LLMs in highly specialized domains such as finance, we need professionally relevant benchmarks that go beyond simple recall and assess complex cognitive skills. Popular benchmarks [11, 18, 14] tend to test general-purpose quantitative or reasoning skills but are not designed to capture nuanced reasoning required for financial analysis, involving synthesis of quantitative data, regulatory knowledge, and market dynamics to make investment decisions [13] . The Chartered Financial Analyst (CFA) Level III exam represents an ideal evaluation framework, being the culminating assessment for investment management professionals. Level III is the final CFA exam, taken after the candidates have passed Levels I and II, and is centered portfolio management and wealth planning, assessing candidates’ ability to implement financial knowledge in real-world scenarios [4, 6] . It uses a mixed format, comprising 11 item set or multiple choice questions (MCQs) and 11 essay or constructed response questions. Each question set begins with a narrative scenario and data (the vignette), which must be carefully analyzed to answer the questions

FAQs

How is CFA Level III used to evaluate advanced financial reasoning in this study?

The study uses mock CFA Level III exams, which include multiple-choice items and constructed-response/essay questions based on narrative scenarios and provided data.

Which prompting strategies are compared for evaluating the LLMs?

The evaluation uses three prompting strategies: zero-shot, chain-of-thought, and self-discover prompting.

What performance differences emerge between multiple-choice and essay questions?

Frontier reasoning models perform similarly on multiple-choice questions, but only a few models excel on the more complex essay questions requiring analysis, synthesis, and strategic thinking.

Sep 4, 2026

Uploaded by Seraphina