CT-FineBench - A Diagnostic Fidelity Benchmark for Fine-Grained Evaluation of CT Report Generation

Aug 20, 2026

Generated radiology report evaluation remains a critical barrier in CT report generation, because outputs contain long text, diverse and complex findings, and fine-grained, disease-oriented attributes. Existing metrics typically measure coarse lexical overlap or limited entity matching, masking clinically required diagnostic fidelity. CT-FineBench builds a QA-based benchmark grounded in CT-RATE and Merlin to score answer correctness for specific clinical attributes such as location, size, margin. Experiments show improved correlation with expert assessment and higher sensitivity to fine-grained factual errors than prior metrics.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages14

Reading Time35 min

AccessFree to download

Introduction

Why diagnostic fidelity is hard to evaluate

Limitations of existing metrics

QA-based evaluation motivation

CT-FineBench

Core idea and contributions

Benchmark construction workflow

Evaluation protocol (QA querying and scoring)

CT-FineBench: A Diagnostic Fidelity Benchmark for Fine-Grained Evaluation of CT Report Generation Ruifeng Yuan1,2,3 , Wanxing Chang1,2 , Weiwei Cao1,2,4 , Bowen Shi1,2,5 , Zhongyu Wei3 , Ling Zhang1 , Jianpeng Zhang1,2,4 1DAMO Academy, Alibaba Group, China, 2Hupan Lab, 310023, China, 3Fudan University, China, 4Zhejiang University, China, 5 Shanghai Jiao Tong University, China Correspondence: [jianpeng.zhang0@gmail.com](jianpeng.zhang0@gmail.com) Abstract The evaluation of generated reports remains a critical challenge in Computed Tomography (CT) report generation, due to the large volume of text, the diversity and complexity of findings, and the presence of fine-grained, disease-oriented attributes. Conventional evaluation metrics offer only coarse measures of lexical overlap or entity matching and fail to reflect the granular diagnostic accuracy required for clinical use. To address this gap, we propose CT-FineBench, a benchmark built from CTRATE and Merlin to evaluate the fine-grained factual consistency of CT reports, constructed from CT-RATE and Merlin. Our benchmark is constructed through a meticulous, QuestionAnswering (QA) based process: first, we identify and structure key, finding-specific clinical attributes (e.g., location, size, margin) . Second, we systematically transform these attributes into a QA dataset, where questions probe for specific clinical details grounded in gold-standard reports. The evaluation protocol for CT-FineBench involves using this QA dataset to query a machine-generated report and scoring the correctness of the answers. This allows for a comprehensive, interpretable, and clinically-relevant assessment, moving beyond superficial lexical overlap to pinpoint specific clinical errors. Experiments show that CTFineBench correlates better with expert clinical assessment and is substantially more sensitive to fine-grained factual errors than prior metrics. 1 Introduction The automatic generation of radiology reports from medical images, particularly Computed Tomography (CT) scans, promises to enhance the efficiency of clinical workflows. However, the clinical adoption of such systems hinges on robust evaluation. For complex and information-dense CT reports, diagnostic fidelity is paramount. This extends beyond the identification of findings to the precise characterization of their clinical attributes. A single flaw in reporting the location, morphology, or severity of a lesion can potentially compromise diagnostic accuracy. Therefore, a fine-grained evaluation metric focusing on clinical attributes is critical for CT report generation. Existing evaluation metrics for radiology report generation can be classified into three types. Conventional linguistic evaluation metrics like ROUGE (Lin, 2004) and BLEU (Papineni et al., 2002), which are based on lexical overlap, are fundamentally inadequate for this task. Even more advanced, embedding-based metrics like BERTScore (Zhang et al., 2019), while better at capturing semantic similarity, still fail to identify and prioritize key medical information. Consequently, all these metrics often assign high scores to reports that are linguistically similar but clinically incorrect. Recognizing this gap, recent research has moved towards more clinically-aware evaluation paradigms. One type of work focuses on entity-based metrics, such as RadGraph (Jain et al., 2021) and RaTEScore (Zhao et al., 2024), which assess reports by extracting and comparing key medical entities like finding/disease and anatomical structures. However, their reliance on a limited set of coarse-grained entity and relation types means they often fail to capture the critical, fine-grained attributes that are crucial for diagnosis, particularly in complex CT reports. Another emerging approach, exemplified by metrics like GREEN (Ostmeier et al., 2024), leverages Large Language Models (LLMs) as judges. Their reliance on general LLMs turns them into black box that offers feedba

FAQs

Why are conventional metrics insufficient for CT report generation evaluation?

They mostly rely on lexical overlap or semantic similarity and cannot reliably detect clinically critical mistakes in fine-grained diagnostic attributes.

How does CT-FineBench evaluate fine-grained factual consistency?

It converts clinical finding attributes into a QA dataset, then queries a generated report and scores whether the answers to attribute-specific questions are correct.

What clinical attributes does CT-FineBench focus on?

It emphasizes fine-grained attributes of findings, such as location, size, density, and margin, rather than only coarse findings.

Aug 20, 2026

Uploaded by Seraphina