Cross-modal deep learning framework for 3D reconstruction and information integration of Zhejiang wood carving heritage - Adaptive cross-modal fusion for heritage reconstruction

Sep 30, 2026

Upload by Miles

A study addresses the limitations of fixed-weight fusion in 3D reconstruction for Zhejiang wood carving heritage by proposing an adaptive cross-modal deep learning framework. The method uses surface-complexity-aware gating networks to dynamically weight geometric and visual modalities according to local informativeness. Evaluation on a newly built dataset of 300 annotated artifacts, captured with hybrid laser scanning and eight-camera RGB-D arrays, reports 0.52 mm Chamfer Distance, 86.7% F-Score, and 76.3% mIoU for semantic segmentation. Results improve over baseline approaches and better preserve intricate openwork and relief structures, enabling precise museum documentation and semantic-rich digitization.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages14

Reading Time35 min

AccessFree to download

Introduction

Neural rendering and multimodal models for heritage

Zhejiang wood carving reconstruction challenges

Proposed method

Surface-complexity-aware adaptive fusion framework

Dataset and evaluation

Newly constructed annotated artifact dataset

Results and implications

Reconstruction accuracy and semantic preservation

Applications for museums and conservation

[www. nature.com/scientificreports](www. nature.com/scientificreports) OPEN Cross-modal deep learning framework for 3D reconstruction and information integration of Zhejiang wood carving heritage Juya Wang & Haifeng Xu􀀍 Current 3D reconstruction methods employ fixed-weight fusion strategies that ignore local surface characteristics in heritage artifacts. This study introduces an adaptive cross-modal deep learning framework for Zhejiang wood carving heritage, featuring surface-complexity-aware gating networks that dynamically weight geometric and visual modalities based on local informativeness. The approach was validated on a newly constructed dataset of 300 annotated artifacts captured using hybrid laser scanning and eight-camera RGB-D arrays. Experimental results demonstrate that the framework achieves 0.52 mm Chamfer Distance and 86.7% F-Score, representing 20% and 6.8% improvements over 3D Gaussian Splatting, while semantic segmentation reaches 76.3% mIoU, surpassing pointcloud-only methods by 11.7%, with particularly superior performance in preserving intricate openwork structures and relief patterns. The adaptive fusion strategy successfully addresses the heterogeneous nature of wood carving surfaces through mathematically principled modality weighting, establishing new benchmarks for heritage reconstruction accuracy. The framework enables museums and conservation institutions to document collections with unprecedented precision while maintaining semantic context, facilitating virtual apprenticeship systems for traditional craft education and intelligent heritage resource management through automated cataloging and quantitative deterioration assessment. Keywords Cross-modal deep learning, 3D reconstruction, Zhejiang wood carving, Cultural heritage preservation, Adaptive fusion mechanism The advent of deep learning has revolutionized cultural heritage preservation, enabling unprecedented capabilities in three-dimensional reconstruction and digital documentation. Neural Radiance Fields (NeRF) area remarkable breakthrough in the field of capturing complex geometric structures with photorealistic quality1. Parallel to this, the recent introduction of 3D Gaussian Splatting has greatly contributed to the viability of realtime rendering in heritage-centric applications2. These neural rendering mechanisms have clearly proven their ability in the reconstruction of architectural heritage, as has been shown through successful reconstructions of diffusion-based cultural heritage3. Aside from these developments in geometry, transformer models have proven to be excellent in multimodal data processing4, with the newer vision transformers showing high interoperability in structural and functional data flows5. Augmented reality interfaces have further extended these possibilities, providing interactive heritage experiences through dual-model approaches integrating depth estimation with neural reconstruction methods, while point cloud processing architectures have evolved from hierarchical feature learning to graph-based and attention-based mechanisms for 3D understanding6. Zhejiang wood carving presents unique reconstruction challenges distinct from general 3D objects: multilayered relief structures with 10–15 cm depth variations create severe self-occlusions; openwork patterns contain hollows as thin as 2–3 mm that confound depth sensors; surface properties vary dramatically from polished lacquer to weathered wood within single artifacts; and dense ornamental motifs require expert knowledge to distinguish primary narrative elements from decorative borders. Unlike conventional 3D reconstruction targets such as buildings with planar facades or manufactured objects with uniform surfaces, wood carvings present heterogeneous material properties where polished lacquer, raw wood, and painted details coexist within centimeter-scale regions, each requiring distinct processing strategies. The computational modeling of multilayered w

FAQs

What problem does the proposed framework address in heritage 3D reconstruction?

Existing methods often use fixed-weight fusion that ignores local surface characteristics, leading to suboptimal reconstruction for heritage artifacts with heterogeneous materials and complex geometry.

How does the framework integrate geometric and visual modalities?

It employs surface-complexity-aware gating networks that dynamically weight geometric and visual modalities based on local informativeness.

What improvements were reported in the experimental results?

The framework achieves 0.52 mm Chamfer Distance and 86.7% F-Score, with about 20% and 6.8% improvements over 3D Gaussian Splatting, and semantic segmentation reaches 76.3% mIoU.

Sep 30, 2026

Upload by Miles