L-MAD A Systematic Evaluation of Multi-Agent Debate Structures in Legal Reasoning

Jul 16, 2026

Upload by Skyler

Multi-agent debate (MAD) frameworks show promise for general reasoning, yet their effectiveness in structured, knowledge-heavy legal settings remains insufficiently studied. This work proposes Legal Multi-Agent Debate (L-MAD) to systematically evaluate debate structures and aggregation methods for Legal Textual Entailment. Distinct expert-persona agents outperform strong single-agent baselines by up to 8%. Scaling analysis reveals trade-offs: more agents reduce inconsistency and improve accuracy, while more rounds cause over-deliberation drift, amplifying errors and harming performance.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages13

Reading Time33 min

Introduction

Preliminaries

Legal Textual Entailment

L-MAD: A Systematic Evaluation of Multi-Agent Debate Structures in Legal Reasoning Tan-Minh Nguyen * 1 Hoang-Trung Nguyen * 2 Huu-Dong Nguyen * 2 Truong Dinh Do 1 Thi-Hai-Yen Vuong 2 Le-Minh Nguyen 1 arXiv :2607 .09099v 1 [ cs .AI] 10 Jul 2026 Abstract While multi-agent debate (MAD) frameworkshave shown significant potential in general reasoning, their effectiveness in highly structured, knowledge-heavy legal domains remains underexplored. In this work, we introduce the Legal Multi-Agent Debate (L-MAD) framework to systematically evaluate different debate structures and aggregation methods within Legal Textual Entailment. By assigning distinct expert personasto multiple agents, L-MAD improves upon strong single-agent baselines by up to 8% . Furthermore, analyzing how debate scales reveals a clear tradeoff: increasing the agent population reduces inconsistency and improves accuracy, whereas extending discussion rounds induces a detrimental over-deliberation drift where agents reinforce each other’s mistakes. Ultimately, our findings outline the practical boundaries and safety margins of deploying collaborative multi-agent systems in high-stakes legal reasoning environments. 1. Introduction Large language models (LLMs) show remarkable capabilities in general reasoning, but they frequently struggle in high-stakes, knowledge-heavy domains that demand strict logic. Legal Textual Entailment (LTE) (Aoki et al., 2022 ; Bilgin et al., 2024) is a challenging test for these models, requiring them to determine whether specific legal statutes apply to a given factual scenario (Aletras et al., 2016 ; Zhong et al., 2020) . While specialized models like LegalBERT (Chalkidis et al., 2020) and Lawformer (Xiao et al., 2021) outperform generalist baselines, standard single-model inference is still limited by a model’s basic reasoning capacity and its tendency to make factual or logical errors. 1Japan Advanced Institute of Science and Technology (JAIST), Ishikawa, Japan 2VNU University of Engineering and Technology, Hanoi, Vietnam. Correspondence to: Le-Minh Nguyen <[nguyenml@jaist.ac.jp](nguyenml@jaist.ac.jp) >. Preprint. July 13, 2026. Figure 1. Illustration of decision-making protocols used in this study. Multi-Agent Debate (MAD) frameworks offer a promising way to overcome these limitations by allowing models to spend more computing power during inference (Nie et al., 2020 ; Choi et al., 2026) . By assigning distinct expert personas to multiple LLM agents, these frameworks let models iterate and refine their arguments to arrive at a better final answer (Kaesberg et al., 2025b) . However, how these debate frameworks perform in highly structured, rule-bound fields like law remains underexplored. While open-ended debate typically helps in general tasks, it introduces unique challenges in legal reasoning. The trade-offs between forcing agents to agree versus letting them vote independently—and the risk of errors multiplying over several rounds—have not yet been systematically analyzed. To bridge this gap, we introduce Legal Multi-Agent Debate (L-MAD), a framework designed to analyze different multiagent debate structures under strict legal rules (Figure 1) . Unlike prior work that mainly focuses on connecting LLMs to external tools, L-MAD provides a controlled environment to evaluate core aggregation methods. Specifically, we compare forced consensus strategies (where agents debate until they agree) against voting approaches (where agents maintain independent reasoning paths) . Our empirical results show that L-MAD significantly improves performance, outperforming strong single-agent baselines by up to 8% in some settings. Crucially, we find that the best debate strategy depends heavily on the underlying model’s capability. Forcing a consensus works well with highly capable models (e.g., 30B+ parameters) by encouraging constructive debate, whereas independent voting protects smaller models (e.g., 8B parameters) from s

FAQs

What does L-MAD aim to evaluate in legal reasoning tasks?

L-MAD systematically evaluates different multi-agent debate structures and aggregation methods for Legal Textual Entailment, focusing on how debate protocols affect decision-making under legal rules.

How does L-MAD perform compared with single-agent baselines?

L-MAD improves performance and can outperform strong single-agent baselines by up to 8% in certain settings.

What scaling trade-off does the document report for L-MAD?

Increasing the number of agents reduces inconsistency and improves accuracy, but increasing the number of discussion rounds induces over-deliberation drift, where agents reinforce each other’s mistakes and performance degrades.

Jul 16, 2026

Upload by Skyler