Automated Parsing of Engineering Drawings for Structured Information Extraction Using a Fine-tuned Document Understanding Transformer

Oct 23, 2026

Accurate extraction of key information from 2D engineering drawings is essential for high-precision manufacturing, yet manual extraction remains slow and labor-intensive and traditional OCR struggles with complex layouts and overlapping symbols. This paper introduces a hybrid deep learning framework that combines oriented bounding-box detection (YOLOv11) with a transformer-based document understanding model (Donut). An in-house annotated dataset trains YOLOv11 to detect nine drawing categories, and cropped detections fine-tune Donut to generate structured JSON outputs. Single-model fine-tuning outperforms category-specific models across evaluation metrics, achieving higher precision (94.77% for GD&T), recall (100% for most categories), and F1 (97.3%), while reducing hallucinations (5.23%).

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages5

Reading Time13 min

AccessFree to download

I. INTRODUCTION

II. METHODOLOGY

Automated Parsing of Engineering Drawings for Structured Information Extraction Using a Fine-tuned Document Understanding Transformer Muhammad Tayyab Khan 1,3\#, Zane Yong2\#, Lequn Chen2, Jun Ming Tan2, Wenhe Feng 1, and Seung Ki Moon3* 1 Singapore Institute of Manufacturing Technology (SIMTech), Agency for Science, Technology and Research (A*STAR), 5 CleanTech Loop, \#01-01 CleanTech Two Block B, Singapore 636732, Republic of Singapore 2 Advanced Remanufacturing and Technology Centre (ARTC), Agency for Science, Technology and Research (A*STAR), 3 CleanTech Loop, \#01-01 CleanTech Two, Singapore 637143, Republic of Singapore 3 School of Mechanical and Aerospace Engineering, Nanyang Technological University, 639798, Singapore * [skmoon@ntu.edu.sg](skmoon@ntu.edu.sg) (S.K. Moon) \# Equal contribution Abstract -Accurate extraction of key information from 2D engineering drawings is crucial for high-precision manufacturing. Manual extraction is slow and laborintensive, while traditional Optical Character Recognition (OCR) techniques often struggle with complex layouts and overlapping symbols, resulting in unstructured outputs. To address these challenges, this paper proposes a novel hybrid deep learning framework for structured information extraction by integrating an Oriented Bounding Box (OBB) detection model with a transformer-based document parsing model (Donut). An in-house annotated dataset is used to train YOLOv11 for detecting nine key categories: Geometric Dimensioning and Tolerancing (GD&T), General Tolerances, Measures, Materials, Notes, Radii, Surface Roughness, Threads, and Title Blocks. Detected OBBs are cropped into images and labeled to fine-tune Donut for structured JSON output. Fine-tuning strategies include a single model trained across all categories and category-specific models. Results show that the single model consistently outperforms categoryspecific ones across all evaluation metrics, achieving higher precision (94.77% for GD&T), recall (100% for most categories), and F1 score (97.3%), while reducing hallucinations (5.23%). The proposed framework improves accuracy, reduces manual effort, and supports scalable deployment in precision-driven industries. Keywords - 2D Engineering Drawings, Deep Learning, Document Understanding Transformer (Donut), FineTuning, Structured Information Extraction, YOLOv11 I. INTRODUCTION Accurate extraction of critical information from 2D engineering drawings is essential for precision manufacturing, directly impacting product quality, efficiency, and cost-effectiveness [1], [2], [3] . These drawings contain technical specifications that guide key decisions, such as manufacturing tool and process selection, which ultimately affect production costs and timelines. Traditionally, this information is extracted manually by experienced engineers who interpret annotations on the drawings [4] . However, manual extraction is time-consuming, error-prone, and especially challenging when dealing with intricate symbols or densely annotated layouts. Misinterpretations can lead to incorrect tooling and machining parameter settings, defective components, and significant production delays [5] . Existing semi-automated approaches, such as ballooning techniques [6] and commercial tools like Mitutoyo’s MeasurLink [7], offer partial automation but still rely heavily on human input. This reliance limits scalability and introduces inconsistencies. Machine learning-based methods, including object detection using YOLO [8] and Optical Character Recognition (OCR) tools [9], have shown promise but face notable challenges. Generic OCR models often produce segmentation errors when handling overlapping annotations, non-standard symbols, or complex layouts. These limitations largely arise from a lack of domain-specific fine-tuning, as most OCR systems are trained on general datasets rather than engineering documents. Khan et al. [5] proposed a fine-tuned OCR-based method tailored to 2

FAQs

What problem does the paper target in extracting information from 2D engineering drawings?

It targets the difficulty of accurately extracting key annotations from complex, densely laid-out drawings, where manual work is slow and error-prone and generic OCR often produces unstructured or incorrect outputs due to overlaps and non-standard symbols.

How does the proposed framework work at a high level?

It uses YOLOv11 OBB detection to locate annotation regions for nine key categories, crops detected regions into labeled image patches, and then fine-tunes a Donut transformer model to produce structured JSON outputs.

What are the main findings from comparing single-model and category-specific fine-tuning?

The single model consistently outperforms category-specific models across evaluation metrics, with reported precision of 94.77% for GD&T, recall of 100% for most categories, F1 of 97.3%, and hallucination reduction to 5.23%.

Oct 23, 2026

Upload by How to eat