Fine-Tuning Vision-Language Model for Automated Engineering Drawing - Information Extraction

Oct 29, 2026

Upload by Jasmine

Geometric Dimensioning and Tolerancing (GD&T) is central to manufacturing quality because it defines allowable variations for part features, yet extracting GD&T from 2D engineering drawings remains slow and labor-intensive. This study fine-tunes Florence-2, an open-source vision-language model, using 400 drawings with ground-truth annotations from domain experts. GPT-4o and Claude-3.5-Sonnet are compared on the same dataset, using zero-shot evaluation and metrics including precision, recall, F1-score, and hallucination rate. Florence-2 shows improved precision, recall, F1-score, and reduced hallucination, demonstrating efficient automation for downstream manufacturing tasks.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages8

Reading Time20 min

AccessFree to download

Abstract

Keywords

1 Introduction

2 Methodology Overview

Fine-Tuning Vision-Language Model for Automated Engineering Drawing Information Extraction Muhammad Tayyab Khan 1, 3, Lequn Chen2, Ye Han Ng2, Wenhe Feng 1, Nicholas Yew Jin Tan 1, Seung Ki Moon3 1 Singapore Institute of Manufacturing Technology (SIMTech), Agency for Science, Technology and Research (A*STAR), 5 CleanTech Loop, \#01-01 CleanTech Two Block B, Singapore 636732, Republic of Singapore 2 Advanced Remanufacturing and Technology Centre (ARTC), Agency for Science, Technology and Research (A*STAR), 3 CleanTech Loop, \#01-01 CleanTech Two, Singapore 637143, Republic of Singapore 3 School of Mechanical and Aerospace Engineering, Nanyang Technological University, 639798, Singapore Abstract Geometric Dimensioning and Tolerancing (GD&T) plays a critical role in manufacturing by defining acceptable variations in part features to ensure component quality and functionality. However, extracting GD&T information from 2D engineering drawings is a time-consuming and labor-intensive task, often relying on manual efforts or semi-automated tools. To address these challenges, this study proposes an automated and computationally efficient GD&T extraction method by fine-tuning Florence-2, an open-source vision-language model (VLM) . The model is trained on a dataset of 400 drawings with ground truth annotations provided by domain experts. For comparison, two state-of-the-art closed-source VLMs, GPT-4o and Claude- 3.5-Sonnet, are evaluated on the same dataset. All models are assessed using precision, recall, F1-score, and hallucination metrics. Due to the computational cost and impracticality of fine-tuning large closed-source VLMs for domain-specific tasks, GPT-4o and Claude-3.5-Sonnet are evaluated in a zero-shot setting. In contrast, Florence-2, a smaller model with 0.23 billion parameters, is optimized through full-parameter fine-tuning across three distinct experiments, each utilizing datasets augmented to different levels. The results show that Florence-2 achieves a 29.95% increase in precision, a 37.75% increase in recall, a 52.40% improvement in F1-score, and a 43.15% reduction in hallucination rate compared to the best-performing closed-source model. These findings highlight the effectiveness of fine-tuning smaller, open-source VLMs like Florence-2, offering a practical and efficient solution for automated GD&T extraction to support downstream manufacturing tasks. Keywords: Geometric Dimensioning and Tolerancing (GD&T), 2D Engineering Drawings, Vision-Language Models, FineTuning, Advanced Manufacturing, Automated Information Extraction. 1 Introduction Geometric Dimensioning and Tolerancing (GD&T) is essential in manufacturing for specifying allowable variations in part features, ensuring that components meet precise functional and quality standards [1] . GD&T uses standardized symbols and rules to clearly communicate geometric requirements on engineering drawings, with Feature Control Frames (FCFs) specifying tolerances, geometric characteristics, and relevant datums. Accurate interpretation of GD&T symbols is crucial for tasks such as inspection, quality control, and assembly, as even minor misinterpretations can lead to production defects and non-conforming parts [2] . Traditionally, extracting GD&T information from 2D engineering drawings has been a manual process , often involving techniques like ballooning, where engineers manually mark and label features [3] . While tools such as Mitutoyo’sMeasurLink [4] assist in these processes, these manual methods remain slow, error-prone, and difficult to scale in high-volume production environments. Additionally, variability in interpretation and data entry errors can result in costly rework and production delays [5] . Recent advancements in machine learning methods, such as object detection algorithms (YOLO [6]) and Optical Character Recognition (OCR) technologies (Tesseract [7]), have been applied to automate GD&T extraction, reducing manual effort. However,

FAQs

What problem does the study address in GD&T extraction from engineering drawings?

The study targets the time-consuming and labor-intensive process of extracting GD&T information from 2D engineering drawings, which often relies on manual or semi-automated approaches.

How is Florence-2 used in the proposed approach?

Florence-2 is fine-tuned using full-parameter fine-tuning on a dataset of 400 drawings with ground-truth annotations, across three experiments with differently augmented datasets.

Which models are compared and what evaluation metrics are used?

The study compares Florence-2 with closed-source VLMs GPT-4o and Claude-3.5-Sonnet in zero-shot settings, evaluating precision, recall, F1-score, and hallucination rate.

Oct 29, 2026

Upload by Jasmine