Hyper-modal Imputation Diffusion Embedding with Dual-Distillation for Federated Multimodal Knowledge Graph Completion

Jul 16, 2026

Upload by Logic

FedMKGC targets training collaborative multimodal knowledge graph completion models across decentralized institutions while preventing sensitive multimodal knowledge transmission. The work introduces MMFeD3-HidE to address uncertain modality unavailability and heterogeneous clients. HidE reconstructs complete multimodal distributions from incomplete entity embeddings using available modalities, while MMFeD3 performs logit and feature dual distillation between clients and server to improve global convergence and semantic consistency. A FedMKGC benchmark and baselines validate the effectiveness, semantic consistency, and convergence robustness.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages16

Reading Time40 min

Introduction

Federated Multimodal Knowledge Graph Completion (FedMKGC)

MMFeD3-HidE framework

Hyper-modal Imputation Diffusion Embedding (HidE)

Multimodal FeDerated Dual Distillation (MMFeD3)

FedMKGC benchmark and experimental validation

Hyper-modal Imputation Diffusion Embedding with Dual-Distillation for Federated Multimodal Knowledge Graph Completion Ying Zhang, Member, IEEE, Yu Zhao, Xuhui Sui, Baohang Zhou, Xiangrui Cai, Li Shen, Member, IEEE, Xiaojie Yuan, Dacheng Tao, Fellow, IEEE arXiv :2506 .22036v2 [ cs .LG] 13 Jul 2026 Abstract—With the increasing multimodal knowledge privatization requirements, multimodal knowledge graphs in different institutes are usually decentralized, lacking of effective collaboration system with both stronger reasoning ability and transmission safety guarantees. In this paper, we propose the Federated Multimodal Knowledge Graph Completion (FedMKGC) task, aiming at training over federated MKGs for better predicting the missing links in clients without sharing sensitive knowledge. We propose a framework named MMFeD3-HidE for addressing multimodal uncertain unavailability and multimodal client heterogeneity challenges of FedMKGC. (1) Inside the clients, our proposed Hyper-modal Imputation Diffusion Embedding model (HidE) recovers the complete multimodal distributions from incomplete entity embeddings constrained by available modalities. (2) Among clients, our proposed Multimodal FeDerated Dual Distillation (MMFeD3) transfers knowledge mutually between clients and the server with logit and feature distillation to improve both global convergence and semantic consistency. We propose a FedMKGC benchmark for a comprehensive evaluation, consisting of a general FedMKGC backbone named MMFedE, datasets with heterogeneous multimodal information, and three groups of constructed baselines. Experiments conducted on our benchmark validate the effectiveness, semantic consistency, and convergence robustness of MMFeD3-HidE. Index Terms—Multimodal Knowledge Graphs, Federated Learning, Multimodal Learning. I. INTRODUCTION Multimodal knowledge graphs (MKGs) [1], [2] organize graph structures composed of relational triples (head entity, relation, tail entity) and their visual and textual attributes as Figure 1, which have been widely in multimodal knowledgeintensive tasks [3]–[6] . Due to incomplete construction and This research was supported by the National Natural Science Foundation of China (No. 62272250, 62572260), the Natural Science Foundation of Tianjin, China (No. 22JCJQJC00150), and the Fundamental Research Funds for the Central Universities, Nankai University (No. 63253232) . The work of Yu Zhao was supported by the China Scholarship Council (No. 202506200065) .(Corresponding author: Yu Zhao.) Ying Zhang, Yu Zhao, Xuhui Sui, Xiangrui Cai, Xiaojie Yuan are with the College of Computer Science, VCIP, DISSec, Nankai University, Tianjin, China (e-mail: [yingzhang@nankai.edu.cn](yingzhang@nankai.edu.cn), [zhaoyu@dbis.nankai.edu.cn](zhaoyu@dbis.nankai.edu.cn), suix [uhui@dbis.nankai.edu.cn](uhui@dbis.nankai.edu.cn), [caixr@nankai.edu.cn](caixr@nankai.edu.cn), [yuanxj@nankai.edu.cn](yuanxj@nankai.edu.cn)). Baohang Zhou is with the School of Software, Tiangong University, Tianjin, China (e-mail: [zhoubaohang@tiangong.edu.cn](zhoubaohang@tiangong.edu.cn)). Li Shen is with the Shenzhen Campus of Sun Yat-sen University, Shenzhen, China ([e-mail: mathshenli@gmail.com](e-mail: mathshenli@gmail.com)). Dacheng Tao is with the College of Computing and Data Science, Nanyang Technological University, Singapore (e-mail: [dacheng.tao@ntu.edu.sg](dacheng.tao@ntu.edu.sg)). ©2026 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. Published in IEEE Transactions on Multimedia. DOI: 10.1109/TMM.2026.3705201 Fig. 1. Toy example of the FedMKGC task for decentralized MKGs. The client MKGs have uncertain unavailabl

FAQs

What problem does FedMKGC address?

FedMKGC trains a global model to complete missing links across decentralized multimodal knowledge graphs while avoiding transmission of sensitive multimodal information.

How does HidE improve multimodal completion on a client?

HidE recovers complete multimodal distributions from incomplete entity embeddings, constrained by modalities that are actually available on the client.

What role does MMFeD3 play in federated learning?

MMFeD3 transfers knowledge between clients and the server via logit and feature distillation, improving global convergence and semantic consistency.

Jul 16, 2026

Upload by Logic