Auto-DSM Under the Lens A Black-Box Evaluation Framework for LLM-Based DSM Generation

Jul 16, 2026

Upload by Oliver

This paper proposes a black-box evaluation framework for assessing how well Large Language Models (LLMs) generate Design Structure Matrices (DSMs) from structured technical documentation. Motivated by the closed nature of existing Auto-DSM pipelines, the framework benchmarks generated DSMs against manually validated ground-truth matrices and supports single-run and multi-run evaluation. It combines structural, classification, and stability measures into a Composite Quality Score. Experiments on a fictive abstract system and a real refrigerator decomposition reveal sensitivity to ambiguity, inconsistent dependency definitions, and prompt formulation, clarifying hallucination and abstention failure modes while enabling auditable Auto-DSM benchmarking for MBSE integration.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages29

Reading Time1 h 13 min

AccessFree to download

Introduction

Problem: Manual and error-prone DSM decomposition

MBSE and structured dependency extraction

LLMs for DSM generation and the need for evaluation

Auto-DSM Under the Lens: A Black-Box Evaluation Framework for LLM-Based DSM Generation N. Potters, T. Hofman arXiv :2607 .05985v 1 [ cs .AI ] 7 Jul 2026 Abstract—This paper presents a black-box evaluation framework to systematically assess the ability of Large Language Models (LLMs) to generate Design Structure Matrices (DSMs) from structured technical documentation. Motivated by the closedsource nature of current Auto-DSM pipelines, the framework introduces a reproducible methodology that benchmarks generated DSMs (GEN-DSMs) against manually validated groundtruth matrices (GT-DSMs). The evaluation integrates both singlerun and multi-run perspectives, combining structural metrics (Completeness, Correctness, Coupling Density), classification metrics (Selective Accuracy, Abstention Coverage), and stability measures (Entropy, Fleiss’ κ). To synthesize these aspects, a Composite Quality Score (Q) is proposed. Controlled experiments are conducted on two datasets: a fictive abstract system anda real-world refrigerator decomposition, covering variations in phrasing, parameterdataset alignment, and system complexity. Results show that LLMs can produce structurally plausible DSMsand achieve high reproducibility under well-structured inputs, but remain sensitive to ambiguity, inconsistent dependency definitions, and prompt formulation. The findings highlight systematic sources of hallucination and abstention failure, demonstrating both the potential and current limitations of LLM-driven DSM automation. The proposed framework provides a transparent benchmark for auditing Auto-DSM pipelines and establishes foundations for integrating LLM-based decomposition methods into model-based systems engineering (MBSE) workflows. Index Terms—Design Structure Matrix (DSM), System decomposition, Large Language Models (LLMs), Black-box evaluation, Model-Based Systems Engineering (MBSE), Dependency analysis, Reproducibility, Automation in engineering design I. INTRODUCTION The growing complexity of engineered systems has driven demand for tools and methods that support system-level analysis and decision-making processes. Among these, the Design Structure Matrix (DSM), introduced by Steward [1], remains a foundational method for visualizing and analyzing interdependencies between system elements. Its compact N × N format supports modularity analysis, design optimization, and process structuring across domains such as product development and systems engineering [2] . Over decades, its use has matured through improvements in visualization and analysis toolse.g., hierarchical clustering and dependency algorithmsforming a robust framework for downstream decision support [2], [3] . The DSM-building phaseidentifying components and interactionsremains manual, resource-intensive, and error-prone N. Potters and T. Hofman (e-mail: [t.hofman@tue.nl](t.hofman@tue.nl)) are with the Eindhoven University of Technology (TU/e), Dept. of Mechanical Engineering, Control Systems Technology section, Engineering Systems Design group, P.O.Box 513, 5600 MB Eindhoven, The Netherlands. [2]–[5] . System decompositions typically rely on expert interviews and informal documentation, introducing variability, inconsistent terminology, and interpretation errors [2], [6], [7] . Poor documentation [8] and tacit organizational knowledge [7] further undermine reproducibility. As noted in [9], interviews are just as important as reading design documents, but they trade speed for accuracy, making DSM system decomposition costly and inconsistent. Model-Based Systems Engineering (MBSE) provides a structured approach to managing system complexity and mitigating these issues [10] . By formalizing specifications in digital models, MBSE enables scalable and consistent dependency mapping [11]–[13] . A recent extension, the Elephant Specification Language (ESL) [14], extracts dependencies from structured natural language, improving traceability and reducing ambiguity. ESL support

FAQs

What does the Auto-DSM black-box evaluation framework measure?

It measures how accurately LLMs generate Design Structure Matrices from technical documentation by comparing generated DSMs to ground-truth matrices and evaluating structure, classification behavior, and stability.

How is evaluation performed in single-run and multi-run settings?

The framework integrates both single-run and multi-run perspectives, using structural metrics, classification metrics such as selective accuracy and abstention coverage, and stability measures like entropy and Fleiss’ κ.

What factors most affect LLM performance in DSM generation?

Results show sensitivity to ambiguity, inconsistent dependency definitions, and prompt formulation, which drive hallucination and abstention failures even when outputs look structurally plausible.

Jul 16, 2026

Upload by Oliver