Measuring the Practice of Shared-Decision Making (OPTION12) An Investigation into Open-sourced Smaller LLMs (OS-sLLMs) for Better Privacy and Sustainability

Jul 16, 2026

Upload by Oliver

Shared decision-making (SDM) in clinical consultations is commonly measured using the OPTION12 instrument, where 12 items are observer-rated on a 5-point Likert scale. Human coding is time-intensive and often suffers from coder disagreement. The study investigates whether open-source privacy-preserving smaller LLMs (OS-sLLMs) can perform OPTION12 scoring with humans in the loop. Using Dutch melanoma consultation transcripts, models are evaluated via prompt refinement, correlation analyses, and judge-LLM-based disagreement resolution.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages14

Reading Time35 min

Background

SDM and OPTION12

Methods

Data and double coding

OS-sLLM deployment and judge-LLM

Preliminary Results

Model comparisons and correlations

Item-level agreement

Impacts

Automation potential and human-in-the-loop deployment

Measuring the practice of shared-decision making (OPTION12): An Investigation into Open-sourced Smaller LLMs (OS-sLLMs) for Better Privacy and Sustainability Tamara Wit 1†, Lifeng Han 1 ,2†, Carly Heipon 1 , David Lindevelt2 Anne Stiggelbout 1 , Suzan Verberne2 On behalf of the 4D PICTURE consortium BDS, Leiden University Medical Centre, NL The Leiden Institute of Advanced Computer Science, LU, NL † co-first, corresponding: {t.wit, l. han} @ lumc. nl arXiv :2607 .06 127v2 [ cs .CL] 9 Jul 2026 Accepted Abstract Background: Shared decision-making (SDM) is important in clinical consultations in which patients discuss and decide on treatment options with clinicians. This SDM process is often coded with the OPTION12 instrument, an observer-based tool, consisting of 12 items, rated on a 5-point Likert scale. This coding is conducted by human coders. Coding is timeintensive and frequently accompanied by disagreement between coders. We explore the capability of open-source privacy-preserving smaller LLMs (OS-sLLMs) to perform the coding task, potentially automating the process, with humans in the loop. Methods: 26 transcripts of Dutch melanoma patients consultations with clinicians were double-coded by two coders. Two human coders resolved their disagreements after independent coding. To evaluate OPTION12 coding using OS-sLLMs, the consultation data was divided into development and testing sets. We designed the complete investigation framework (see Figure 1) . It includes 1) a pilot study of the development set (11 interviews) for prompt-refinement and sLLM selection. 2) deployment of the fine-tuned prompts and best performing sLLM (judge-sLLM) augmented with few-shot examples from the development set on the testing set (15 interviews) and asking the judgesLLM to resolve the disagreement on other OS-sLLMs’ scores. The rationale for judge-llm: it behaves like a third-party human to resolve the disagreements of annotators. Instead of using humans, a judge-LLM is consulted to resolve disagreement. This is designed to mimic how human annotators resolve disagreements, by discussing until consensus is achieved. We plan to investigate alternative model-merging methods for the future. First, for prompt re- finement, we use chain-of-thoughts (CoTs), LLM-assisted prompting, human-in-the-loop with sample output (few-shots) feedback. For OS-sLLMs, we use both 1) general domain models Llama, Gemma, and Mistral7b, and 2) medical domain models Meditron and Medllama. Second, for judge-sLLM selection, on the system-level, we measure the overall correlation of each sLLM and human coding using Spearman and Pearson correlation scores. On the segment-level, we also look into the most agreed-upon and disagreed-upon items . We will perform both qualitative and quantitative analysis by discussing the evaluation scoresand categorise the OS-sLLMs’ behaviours with examples. Finally, we deploy the OS-sLLMs on the testing set and use the judge-sLLM to resolve disagreements. Preliminary results: Five OS-sLLMs show the following findings: • Three general domain OS-sLLMs perform better than the two medical domain ones, which both generate hallucinations and do not follow prompts precisely, indicating further developments is needed for medical OS-sLLMs. • Mistral7b outperformed the other two Gemma3:12b and Llama3 . 1:8b by 4 consensus with human coding, vs 3 items. • The overall correlation with human coding on these 12 items is (0.83, 0.80, 0.64) using Pearson correlation, and (0.81, 0.78, 0 .61) using Spearman rank correlation from the three models (gemma3:12b, llama3.1:8b, mistral7b) . • For the items for which OS-sLLMs agree with human coders, OS-sLLMs can generate the same sentences as humans in some cases, but in other cases, they generate better quotes than humans. Impacts: This study reports the first research findings using OS-sLLMs on scoring SDM with the OPION12 in Dutch melanoma patients’ consultation transcripts. The perform

FAQs

What problem does the OPTION12 coding process face in practice?

OPTION12 coding relies on human observers and is time-intensive. It also frequently involves disagreements between coders, motivating automated support.

How does the study use open-source smaller LLMs to score SDM?

The work refines prompts and selects OS-sLLMs on a development set, then deploys the best-performing model with few-shot examples on a testing set. A judge-LLM resolves disagreements to mimic third-party human adjudication.

What do preliminary results indicate about which models perform best?

Three general-domain OS-sLLMs outperform two medical-domain models that produce hallucinations and fail to follow prompts precisely. Mistral7b shows the strongest consensus with human coding among the compared models, with high overall correlation on the 12 items.

Jul 16, 2026

Upload by Oliver