Measuring Visual Understanding in Telecom domain - Performance Metrics for Image-to-UML conversion using VLMs

Oct 26, 2026

Upload by Fans

Telecom domain 3GPP documents contain sequence diagrams embedded as images, and Vision-Language Large Models (VLMs) enable conversion into machine-readable PlantUML (puml) scripts. This work introduces evaluation performance metrics to quantify conversion effectiveness across participant identification, message-flow accuracy, sequence ordering, and grouping-construct preservation. A representative dataset of 3GPP sequence diagrams is used, comparing puml outputs from Claude Sonnet and GPT-4V with manually created ground truth. Results show accurate capture of nodes, edges, and messages, while complex structures like notes, boxes, and groups remain challenging, motivating improved representation learning.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages12

Reading Time30 min

AccessFree to download

Abstract

1 Introduction

Measuring Visual Understanding in Telecom domain: Performance Metrics for Image-to-UML conversion using VLMs H. G. Ranjani Ericsson R & D, Bangalore, India [ranjani.h.g@ericsson.com](ranjani.h.g@ericsson.com) Rutuja Prabhudesai IIITB, Bangalore, India [rutuja10feb1999@gmail.com](rutuja10feb1999@gmail.com) Abstract Telecom domain 3GPP documents are replete with images containing sequence diagrams. Advances in Vision-Language Large Models (VLMs) have eased conversion of such images to machine-readable PlantUML (puml) formats. However, there is a gap in evaluation of such conversions-existing works do not compare puml scripts for various components. In this work, we propose performance metrics to measure the effectiveness of such conversions. A dataset of sequence diagrams from 3GPP documents is chosen to be representative of domain-specific actual scenarios. We compare puml outputs from two VLMs—Claude Sonnet and GPT-4V—against manually created ground truth representations. We use version control tools to capture differences and introduce standard performance metricsto measure accuracies along various components: participant identification, message flow accuracy, sequence ordering, and grouping construct preservation. We demonstrate effectiveness of proposed metrics in quantifying conversion errors across various components of puml scripts. The results show that nodes, edges and messages are accurately captured. However, we observe that VLMs do not necessarily perform well on complex structures such as notes, box, groups. Our experiments and performance metrics indicates a need for better representation of these components in training data for fine-tuned VLMs. 1 Introduction Sequence diagrams are widely used to represent signaling sequences and interactions among system components. However, these diagrams are often available only as static images within technical documents and scattered across versions and sections. This limits machine-readability of such sequences and their usability in tools that support analysis, simulation, automated verification and/or troubleshooting. We consider the telecom domain as a case in point to illustrate some challenges using 3rd Generation Partnership Project (3GPP) specifications (3GPP, 2022) . These are publicly available as word documents containing text, tables, equations and images (Roychowdhury et al., 2024a,b, 2025) including sequence diagrams (as images) within to illustrate procedural call flows across various network entities in various scenarios. Recent advances in Vision-Language Large Models (VLMs) have enabled the extraction of structured information from images, including charts, tables and UML (Unified Modeling Language) diagrams. Several studies have proposed methods to extract UML components from visual representations, converting image-based diagrams into machine-readable formats using tools such as PlantUML to create puml scripts (PlantUML, 2025 ; PlantUMLGuide, 2023 ; Romeo et al., 2025) . These approaches aid towards automating the conversion of legacy diagram archives into usable data. In this work, we use puml to refer to UML scripts (accessed or generated using PlantUML tools or equivalent ones) . The work in (De Bari, 2024) uses LLMs to generate UML class diagrams. The diagrams are analyzed for syntactic, semantic, and pragmatic quality against that of human generated UML diagrams. In (Ye et al., 2024), flowchart images are converted to graphical structures using VLMs. Then, these structures are compared for the optimal representation format (puml, Mermaid or Graphviz) for improved performance in reasoning based question answering (QA) task. For evaluating the UML representation format, node-F1 and edge-F1 metrics have been considered. The work of (Axt, 2023) converts human sketches into UML diagrams using OpenCV libraries. The UML diagrams are evaluated based on precision and recall of classes, inheritances, and associations. The work (Conrardy and

FAQs

What problem does the paper address in telecom image-to-UML conversion?

It addresses the lack of systematic evaluation for converting image-based sequence diagrams in telecom (3GPP) documents into PlantUML (puml) scripts.

Which VLMs are evaluated against ground truth representations?

The paper compares puml outputs from Claude Sonnet and GPT-4V with manually created ground truth representations.

What kinds of structures do the proposed metrics measure, and what is the main limitation observed?

The metrics assess participant identification, message-flow accuracy, sequence ordering, and grouping-construct preservation; the experiments show weaker performance on complex structures such as notes, boxes, and groups.

Oct 26, 2026

Upload by Fans