From Voting to Agent Collaboration Answer-Type-Aware LLM Pipelines for BioASQ 14b

Jul 16, 2026

Upload by Ezra

Biomedical question answering must extract reliable information from scientific literature and integrate evidence across multiple documents with strong grounding. This study proposes a question-type-specific LLM framework for BioASQ 14b Task B, improving answer robustness by using distinct inference procedures for yes/no, factoid, and list questions. It applies snippet shuffling and self-reflection for stable binary decisions, full-snippet chain-of-thought in-context learning for biomedical entity identification, and a multi-agent pipeline for evidence extraction, verification, and final aggregation. Evaluations on the official challenge show competitive results and a first-place finish in the factoid subtask of Batch 4.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages15

Reading Time38 min

Abstract

Introduction

Question types in BioASQ Task B

Challenges for biomedical LLMs

From Voting to Agent Collaboration: Answer-Type-Aware LLM Pipelines for BioASQ 14b Taeyun Roh1,†, Eunha Lee1,†, Wonjune Jang2,†, Sohyun Chung1,†, Junha Jung1,† and Jaewoo Kang1,3, * 1 Department of Computer Science and Engineering, Korea University, Seoul, 02841, Republic of Korea 2 Department of Mathematics, Myongji University, Yongin, 17058, Republic of Korea 3 AIGEN Sciences, Seoul, 04778, Republic of Korea Abstract Biomedical question answering requires not only accurate extraction of information from scientific literature but also reliable integration of evidence across multiple documents. This study presents a question-type-specific large language model (LLM) framework for BioASQ 14b Task B, designed to improve answer robustness and evidence grounding in biomedical question answering. Rather than applying a single prompting strategy to all questions, the framework selects different inference procedures for yes/no, factoid, and list questions according to their distinct reasoning and evaluation requirements. For yes/no questions, snippet shuffling and self-reflection are used to reduce sensitivity to evidence ordering and improve decision stability. For factoid questions, full-snippet input is combined with chain-of-thought-based in-context learning to support accurate biomedical entity identification. For list questions, a multi-agent architecture is employed, in which evidence extraction, candidate generation, answer verification, and final aggregation are handled collaboratively. Preliminary experiments on BioASQ 13b were used to identify effective inference strategies for each question type, and the resulting framework was subsequently evaluated in the official BioASQ 14b Task B challenge. In the official evaluation, our framework showed competitive performance across multiple batches and achieved first place in the factoid subtask of Batch 4 . These results demonstrate the effectiveness of combining question-type-specific inference, ensemble prediction, and agent-based verification for reliable biomedical question answering. Keywords BioASQ 14b, LLM, Multi-Agent System, Chain-of-Thought, Prompt Engineering, 1. Introduction Biomedical literature is expanding at a pace that makes it increasingly difficult for researchers and healthcare professionals to identify precise and reliable answers to specialized questions. Important evidence is often distributed across multiple articles, expressed using heterogeneous terminology, or reported in partially complementary and sometimes conflicting contexts. As a result, biomedical QA systems cannot rely on document retrieval alone. They must read evidence in context, connect findings scattered across snippets, and return answers that are concise enough for evaluation while still being grounded in the source literature. The BioASQ challenge has served as a major benchmark for advancing such capabilities by evaluating systems on biomedical semantic indexing and question answering tasks [1, 2] . In particular, BioASQ Task B requires systems to generate exact and ideal answers from evidence snippets extracted from biomedical literature, making it a suitable testbed for evaluating evidence-grounded reasoning in large language models (LLMs) [2, 3] . BioASQ Task B includes several answer formats, among which yes/no, factoid, and list questions pose distinct reasoning challenges. Yes/no questions require a stable binary decision based on potentially CLEF 2026 Working Notes, 21 – 24 September 2026, Jena, Germany * Corresponding author. † These authors contributed equally to this work. $ [nrbsld@korea.ac.kr](nrbsld@korea.ac.kr) (T. Roh); [eunhalee@korea.ac.kr](eunhalee@korea.ac.kr) (E. Lee); [dnjswnswkd03@mju.ac.kr](dnjswnswkd03@mju.ac.kr) (W. Jang); [sohyunjung@korea.ac.kr](sohyunjung@korea.ac.kr) (S. Chung); [goodjungjun@korea.ac.kr](goodjungjun@korea.ac.kr) (J. Jung); [kangj@korea.ac.kr](kangj@korea.ac.kr) (J. Kang) 􀀚 0009-0009-7096-6410 (T. Roh); 0009-0

FAQs

What is the core idea of the proposed framework for BioASQ 14b Task B?

The framework assigns different inference procedures to different question types—yes/no, factoid, and list—to better match their distinct reasoning and evaluation needs and to improve evidence grounding.

How does the framework handle yes/no questions?

It uses snippet shuffling and self-reflection to reduce sensitivity to evidence ordering and to improve decision stability for binary outputs.

What does the framework do for list questions?

It employs a multi-agent architecture where evidence extraction, candidate generation, answer verification, and final aggregation are performed collaboratively to improve coverage while avoiding unsupported or redundant predictions.

Jul 16, 2026

Upload by Ezra