Time Imprint Learning Time-Aware Representations in Multi-Modal Knowledge Graphs

Jul 16, 2026

Upload by Violet

Multi-Modal Knowledge Graphs enrich entities with text and images, yet entities with highly similar multi-modal features remain difficult to distinguish. Temporal information can disambiguate such entities, but prior work seldom treats time as an explicit modality because temporal semantics are sparse and many timestamps add noise. Time Imprint models time as an entity-level modality, jointly aligning temporal, textual, and visual representations using a three-view contrastive objective. Experiments on three benchmarks show state-of-the-art link prediction, up to 6.07% overall Hits@1 improvement, and up to 58% gains for highly ambiguous samples, with only modest training overhead.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages11

Reading Time28 min

Introduction

Motivation and problem of multimodal ambiguity

Limitations of prior time modeling

Method (Time Imprint)

Time as an entity-level modality

Three-view contrastive alignment

Timestamp subset selection and attention pooling

Experiments

Benchmark results and improvements

Analysis of fusion strategies and timestamp quality

Conclusion

When time-as-modality is most beneficial

Time Imprint: Learning Time-Aware Representations in Multi-Modal Knowledge Graphs Pengyu Zhang [p.zhang@uva.nl](p.zhang@uva.nl)[ ](p.zhang@uva.nl)University of Amsterdam Amsterdam, The Netherlands Klim Zaporojets Aarhus University Aarhus, Denmark Congfeng Cao University of Amsterdam Amsterdam, The Netherlands Jia-Hong Huang University of Amsterdam Amsterdam, The Netherlands Paul Groth University of Amsterdam Amsterdam, The Netherlands arXiv :2607 .09777v 1 [ cs .CV] 8 Jul 2026 Abstract Multi-Modal Knowledge Graphs (MMKGs) enrich entities with multiple modalities such as text and images, yet entities with highly similar multi-modal features remain difficult to distinguish. Temporal information of an entity can serve as an additional modality to disambiguate such entities, but existing approaches rarely treat time as a separate modality alongside text and images due to two major challenges: (1) sparse temporal semantics, which hinder alignment with richer modalities, and (2) multiple timestamps, which introduce noise or reduce robustness in representation learning. To address these challenges, we propose Time Imprint, a framework that treats time as an entity-level modality and jointly aligns temporal, textual, and visual representations via a three-view contrastive objective. Additionally, to mitigate multi-timestamp ambiguity, Time Imprint studies a compact timestamp subset selection design space and aggregates the selected timestamps into a discriminative temporal embedding with attention pooling, balancing temporal specificity and robustness. Experiments on three MMKG benchmarks demonstrate that Time Imprint achieves state-ofthe-art link prediction performance, improving Hits@1 by up to 6.07% overall and yielding up to 58% gains on the subset of the top-1% ambiguity samples. We further examine different fusion strategies and the sensitivity to timestamp availability and quality, clarifying when and why time-as-modality is most beneficial, while adding only modest training overhead. We release our code at [https://anonymous.4open.science/r/Time-Imprint](https://anonymous.4open.science/r/Time-Imprint). CCS Concepts • Computing methodologies → Knowledge representation and reasoning; • Information systems → Multimedia and multimodal retrieval; Temporal data. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission [and/or a fee. Request permissions from permissions@acm.org](and/or a fee. Request permissions from permissions@acm.org). Conference acronym ’XX, Woodstock, NY © 2018 Copyright held by the owner/author(s) . Publication rights licensed to ACM. ACM ISBN 978-1-4503-XXXX-X/2018/06 [https://doi.org/XXXXXXX.XXXXXXX](https://doi.org/XXXXXXX.XXXXXXX) Keywords Multi-modal Knowledge Graphs, Temporal Representation Learning, Contrastive Learning, Modality Fusion, Link Prediction ACM Reference Format: Pengyu Zhang, Klim Zaporojets, Congfeng Cao, Jia-Hong Huang, and Paul Groth. 2018. Time Imprint: Learning Time-Aware Representations in MultiModal Knowledge Graphs. In Proceedings of Make sure to enter the correct conference title from your rights confirmation email (Conference acronym ’XX) . ACM, New York, NY, USA, 11 pages. [https://doi.org/XXXXXXX.XXXXXXX](https://doi.org/XXXXXXX.XXXXXXX) 1 Introduction Multi-Modal Knowledge Graphs (MMKGs) enrich entity representations by integrating text, images, audio, and video with structured triples and are widely used in Web applications such as recommendation and search [17] . For ex

FAQs

Why are multi-modal entities difficult to distinguish in multi-modal knowledge graphs?

Entities can share highly similar text and visual content, causing ambiguity that harms representation learning and link prediction.

What major challenges prevent using time as a separate modality alongside text and images?

Temporal semantics are sparse, making alignment with richer modalities difficult; additionally, multiple timestamps introduce noise and reduce robustness.

How does Time Imprint improve link prediction using time information?

It treats time as an entity-level modality and jointly aligns temporal, textual, and visual embeddings with a three-view contrastive objective, while selecting compact timestamp subsets and aggregating them into a discriminative temporal embedding via attention pooling.

Jul 16, 2026

Upload by Violet