MAX-EVAL-11 - A Large-Scale Benchmark for Evaluating Large Language Models on Full-Spectrum ICD-11 Medical Coding

Sep 17, 2026

Uploaded by Mack Lee

The global transition to ICD-11 requires reliable automated medical coding, yet large-scale evaluation resources for full-spectrum ICD-11 were missing. MAX-EVAL-11 introduces a foundational benchmark with 10,000 MIMIC-III discharge summaries and expert-validated ICD-11 annotations covering 99.87% of the diagnostic taxonomy. A hierarchical evaluation framework assigns partial credit using ICD-11’s five-level structure, mitigating brittleness of exact-match metrics. Experiments on a stratified 500-patient sample show low exact-match performance (0–4.8%) and strong dependence on hierarchical scoring, highlighting task difficulty.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages10

Reading Time25 min

AccessFree to download

Abstract

Introduction

Dataset

Related Work

Deep Learning Approaches for ICD Coding

MAX-EVAL-11: A Large-Scale Benchmark for Evaluating Large Language Models on Full-Spectrum ICD-11 Medical Coding Ujjwal Singh Max Healthcare [ujjwal.singh@maxhealthcare.com](ujjwal.singh@maxhealthcare.com) Nitish Dube Max Healthcare [ntish.dube@maxhealthcare.com](ntish.dube@maxhealthcare.com) Sarthak Deshwal Plaid Inc. [sarthakdeshwal@duck.com](sarthakdeshwal@duck.com) Arjun Sharma Max Healthcare [arjun.sharma@maxhealthcare.com](arjun.sharma@maxhealthcare.com) Abstract The global transition to the ICD-11 taxonomy demands robust automated medical coding, yet comprehensive benchmarks to evaluate Large Language Models (LLMs) on this task remain absent. We introduce MAX-EVAL-11 1 asa foundational resource, the first large-scale benchmark for full-spectrum ICD-11 medical coding. MAX-EVAL-11 comprises 10,000 MIMIC-III discharge summaries with mapped, expert-validated subset (12%) ICD-11 annotations spanning 99.87% of the diagnostic taxonomy. To better reflect clinical utility, we propose a novel hierarchical evaluation framework that assigns partial credit based on ICD-11’s 5-level structure, addressing the brittleness of traditional exact-match metrics. To illustrate benchmark utility, we evaluate LLMs on a stratified 500-patient sample, revealing significant performance gaps. The bestperforming model (Claude 4 Sonnet) achievesa weighted score of 0.433, outperforming both general-purpose peers and specialized medical models (MedCoder) . Crucially, all models exhibit near-zero exact match rates (0–4.8%) and rely primarily on hierarchical credit, underscoring the extreme difficulty of precise ICD-11 code generation. Furthermore, the superiority of general-purpose LLMs over legacy ICD-10 medical models (with ICD-11 codelist) suggests that broad reasoning capabilities currently outweigh domain-specific training for complex taxonomy scaling. 1 Introduction Medical coding, transforming clinical narratives into standardized diagnostic codes is fundamental to modern healthcare for statistical analysis, reimbursement, and epidemiological research. The International Classification of Diseases (ICD), maintained by the WHO, provides the global standard for this task. 1Dataset: [https://huggingface.co/datasets/](https://huggingface.co/datasets/)[ ](https://huggingface.co/datasets/)[mas-namtla/max-eval-11](mas-namtla/max-eval-11) (gated access) . ICD-11, officially adopted in 2019 and effective since January 2022, represents a major evolution in medical classification (World Health Organization, 2019) . As of 2024, 132 member states are implementing ICD-11, which introduces over 55,000 unique codes with post-coordination for granular clinical documentation (Harrison et al., 2021);(World Health Organization, 2024) . Manual medical coding faces substantial challenges from growing documentation volumes, evolving regulations, and workforce shortages. The ICD-11 transition intensifies these pressures, requiring complex crosswalk mapping. Only 23.5% of ICD-9 codes achieve direct one-to-one ICD-11 mapping, complicating automated transition (Fung et al., 2021) . Recent AI advances, particularly large language models (LLMs), have sparked interest in automated medical coding. However, specialized models achieve F1 scores of only 0.539 to 0.719 on ICD-10 benchmarks (Baksi et al., 2025 ; Edin et al., 2023), while general-purpose LLMs face additional challenges (Hou et al., 2025) . ICD-11’s multidimensional taxonomy—capable of generating vast combinations via post-coordination—presents even greater complexity. Despite progress, existing benchmarks primarily focus on ICD-9 and ICD- 10. MIMIC-III (Johnson et al., 2016) and MIMICIV-ICD (Nguyen et al., 2023) provide robust evaluation for legacy systems, but no comprehensive ICD-11 benchmark exists. Furthermore, traditional precision, recall, and F1 metrics inadequately capture the hierarchy of clinical relevance: accurate identification of a primary diagnosis is more criti

FAQs

What is MAX-EVAL-11 and what does it evaluate?

MAX-EVAL-11 is a large-scale benchmark for evaluating Large Language Models on full-spectrum ICD-11 medical coding, using MIMIC-III discharge summaries with expert-validated ICD-11 annotations.

How does MAX-EVAL-11 score predictions in evaluation?

It proposes a hierarchical evaluation framework that assigns partial credit based on ICD-11’s five-level structure, instead of relying on brittle exact-match metrics.

What do the benchmark results show about model performance?

All evaluated models show near-zero exact match rates (0–4.8%), relying primarily on hierarchical credit. The best model reported achieves a weighted score of 0.433.

Sep 17, 2026

Uploaded by Mack Lee