Sensemaking in Multi-Agent LLM Interfaces - How Users Interpret Transparency and Trustworthiness Cues

Sep 4, 2026

Uploaded by James

Multi-agent Large Language Models gain traction, but designers must shape how internal reasoning is surfaced to foster appropriate user trust. This design-led qualitative comparative observation study examines how users interpret and evaluate transparency in multi-agent LLM interfaces. Participants interacted with five interface variants across information-seeking and logical-reasoning tasks, forming mental models from transparency and trustworthiness cues. Findings reveal a dynamic “Goldilocks” transparency level shaped by task demands, affordances, expertise, and dispositional AI trust, alongside tensions among visibility, sufficiency, and cognitive effort.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages20

Reading Time50 min

AccessFree to download

Introduction

Background and motivation

Multi-agent LLM interfaces and transparency

Sensemaking in Multi-Agent LLM Interfaces: How Users Interpret Transparency and Trustworthiness Cues Saumya Pareek School of Computing and Information Systems University of Melbourne Melbourne, Victoria, Australia [saumya.pareek@student.unimelb.edu.au](saumya.pareek@student.unimelb.edu.au) Jarod Govers School of Computing and Information Systems University of Melbourne Melbourne, Victoria, Australia [jarod.govers@unimelb.edu.au](jarod.govers@unimelb.edu.au) Naja Kathrine Kollerup Department of Computer Science Aalborg University, Aalborg Denmark [nkka@cs.aau.dk](nkka@cs.aau.dk) Emily Wong School of Computing and Information Systems University of Melbourne Melbourne, Victoria, Australia [emily.wong.1@unimelb.edu.au](emily.wong.1@unimelb.edu.au) Eduardo Velloso School of Computer Science The University of Sydney Sydney, New South Wales, Australia [eduardo.velloso@sydney.edu.au](eduardo.velloso@sydney.edu.au) Jorge Goncalves School of Computing and Information Systems University of Melbourne Melbourne, Victoria, Australia [jorge.goncalves@unimelb.edu.au](jorge.goncalves@unimelb.edu.au) Figure 1: Diagrammatic overview of the five multi-agent Large Language Model (LLM) interface variants (V1–V5) used as exploratory probes in our study. Each variant instantiates a distinct combination of transparency-related design dimensions to surface multi-agent reasoning to the user. V1 (Final Answer) shows only the system’s final decision, with no agent-level visibility. V2 (Agents Answer) displays each agent’s individual answer alongside the majority outcome. V3 (Agents Explain) extends this by surfacing a brief rationale from each agent, before presenting the majority decision. V4 (Agents Critique) introduces a dedicated critic agent who evaluates each agent’s response, followed by a summarised final answer. V5 (Agents Debate) presents a multi-turn inter-agent discussion, where initially disagreeing agents deliberate and converge on a consensus answer. Together, these variants span a range of process visibility and aggregation mechanisms found in emerging multi-agent LLM systems. We examine how these differences shape user perceptions of transparency and trustworthiness, and the tensions that arise between increased system visibility and its cognitive costs, across different task types. Abstract As multi-agent Large Language Models (LLMs) gain traction, designers must consider how to surface their internal reasoning in ways that foster appropriate trust. We present a design-led, qualitative, This work is licensed under a Creative Commons Attribution 4.0 International License. CHI’26, Barcelona, Spain © 2026 Copyright held by the owner/author(s) . ACM ISBN 979-8-4007-2278-3/26/04 [https://doi.org/10.1145/3772318.3791157](https://doi.org/10.1145/3772318.3791157) comparative structured observation study, exploring how users interpret and evaluate transparency in multi-agent LLMs. Participants interacted with five interface variants, each instantiating different combinations of transparency-related design dimensions, across two task types: information-seeking and logical reasoning. We surface participants’ mental models, the cues they interpret as signals of transparency and trustworthiness, and how they weigh the costs and benefits of increasing process visibility. Transparency needs were dynamic and context-sensitive, with the ideal “Goldilocks”(i.e.,“just right” transparency) level shaped jointly by task demands, interface affordances, and user characteristics such as task expertise and dispositional AI trust. We highlight tensions between process visibility, information sufficiency, and cognitive effort, and synthesise these insights into design considerations for aligning transparency with user needs in future multi-agent LLM interfaces. CCS Concepts • Human-centered computing → HCI theory, concepts and models; Collaborative interaction. Keywords transparency, trust, multi-agent chatbots, mul

FAQs

What question does the study investigate about multi-agent LLM interfaces?

The study investigates how users interpret and evaluate transparency and trustworthiness cues in multi-agent LLM interfaces, and how different visibility choices affect user perceptions and cognitive tradeoffs.

How are transparency conditions operationalized in the study?

The study uses five interface variants (V1–V5) that differ in process visibility, including showing only the final answer, showing agents’ answers, adding rationales, introducing a critic agent, or enabling multi-turn agent debate.

What factors determine the study’s “Goldilocks” transparency level?

The ideal transparency level is shaped by task demands, interface affordances, and user characteristics such as task expertise and dispositional AI trust.

Sep 4, 2026

Uploaded by James