Pseudo Contrastive Learning for Diagram Comprehension in Multimodal Models - WACV 2026 paper

Oct 12, 2026

Uploaded by lugxt

Recent multimodal models such as Contrastive Language-Image Pre-training (CLIP) align visual and linguistic representations, yet they remain weak on diagram understanding where small visual changes can shift semantics. This work introduces a pseudo contrastive training paradigm for vision-language models, generating synthetic diagrams via a diagram renderer using randomly selected text elements. Pseudo contrastive samples are incorporated into the training objective to emphasize structural differences without modifying original data. Experiments on flowchart benchmarks show substantial gains in image-text matching and visual question answering versus standard CLIP and hard-negative CLIP.

LanguageEnglish

CategoryGeneral

FormatPDF

Pages9

Reading Time3 min

AccessFree to download

Introduction

Vision-Language Models and CLIP

Why diagrams remain challenging

Limitations of conventional fine-tuning and contrastive learning

Hard negative sampling and its gaps

Rule-based editing for structural control

Conceptual overview

This WACV workshop paper is the Open Access version, provided by the Computer Vision Foundation. Except for this watermark, it is identical to the accepted version; the final published version of the proceedings is available on IEEE Xplore. Pseudo Contrastive Learning for Diagram Comprehension in Multimodal Models Hiroshi Sasaki The Japan Research Institute, Limited [sasaki.hiroshi@jri.co.jp](sasaki.hiroshi@jri.co.jp) Abstract Recent multimodal models such as Contrastive LanguageImage Pre-training (CLIP) have shown remarkable ability to align visual and linguistic representations. However, domains where small visual differences carry large semantic significance, such as diagram understanding, remain challenging due to the models’ limited sensitivity to fine-grained structural variations. We propose a new training paradigm designed to enhance diagram comprehension in vision-language models. Our approach introduces pseudo contrastive samples generated by a diagram renderer that creates synthetic diagrams using randomly picked text elements. These samples highlight structural differences in diagrammatic imagery without requiring any modification or editing of the original data. By incorporating these pseudo contrastive samples into the training objective, the model learns to capture more precise and semantically consistent diagram structures. Empirical evaluations on a benchmark dataset offlowcharts demonstrate substantial improvements over standard CLIP and hard-negative CLIP training in both image-text matching and visual question answering tasks. The results underscore the value of domain-specific training strategies and contribute to advancing diagrammatic understanding within the broader context of vision-language learning. 1. Introduction Vision-Language Models (VLMs), such as CLIP (Contrastive Language-Image Pre-training) [16], have emerged as powerful tools, integrating visual and linguistic information to achieve remarkable capabilities in image understanding. These models have demonstrated superior performance across diverse tasks, ranging from image captioning and visual question answering (VQA) to zero-shot recognition, by effectively bridging the modalities of visual perception and language. Their ability to link visual and textual inputs has Pseude Diagrams (editable format) Hard Positive / Negative Samples Contrastive Learning with Pseude Hard Samples Figure 1 . Conceptual overview of our proposed method. Text is extracted from a raster image via OCR to generate pseudo diagrams with random connections. These are rendered into an editable format, from which hard positive and negative pairs are created through rule-based edits. The resulting samples are used to train a VLM with structure-aware contrastive learning, enhancing its ability to distinguish fine-grained structural differences. advanced artificial intelligence applications. Meanwhile, diagrams, exemplified by flowcharts, serve as indispensable visual representations for conveying complex concepts and intricate procedures that are challenging to express concisely through natural language alone. These visual artifacts are widely employed across various domains, including engineering, software development, business process modelling, and education, due to their clarity and efficacy in illustrating relationships, sequences, and decision points. The inherent value of diagrams underscores the significance of enabling VLMs to comprehend their detailed content. However, despite the general advancements in VLM capabilities, conventional models exhibit a critical shortcoming when it comes to understanding the fine-grained structural information present in diagram images. Structural elements, such as the directional arrows indicating process flow, lines representing connections, or symbols denoting branching conditions between nodes, carry immense semantic weight. Current VLMs often struggle to accurately capture these subtle ye

FAQs

What problem does the paper address in multimodal diagram understanding?

Conventional vision-language models struggle to capture fine-grained structural details in diagram images. Even small structural changes like arrows, connections, or branching conditions can alter meaning, reducing reliability for precise diagram comprehension.

How does the proposed pseudo contrastive learning approach create training samples?

The method generates pseudo diagrams using a diagram renderer with randomly selected text elements. It then creates hard positive and negative pairs via rule-based edits, using pseudo samples to train structure-aware contrastive learning without altering the original data.

What benefits do the experiments report on flowchart benchmarks?

Experiments on a flowchart benchmark show substantial improvements over standard CLIP and hard-negative CLIP. Gains are reported for both image-text matching and visual question answering tasks.

Oct 12, 2026

Uploaded by lugxt