SRCB at SemEval-2022 Task 5 - Pretraining Based Image to Text Late Sequential Fusion System for Multimodal Misogynous Meme Identification

Aug 26, 2026

Upload by Aria

Online misogyny meme detection is a multimodal classification problem where the relationship between image and text strongly affects fusion learning. This work evaluates single-stream UNITER and dual-stream CLIP multimodal pretrained models for handling both strongly and weakly correlated image-text pairs. An XGBoost classifier using CLIP image features achieves the best overall performance and remains robust under domain shift. Building on this, the proposed PBR system ensembles pretrained models, boosting, and rule-based adjustment, integrating text through a late sequential fusion scheme.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages12

Reading Time30 min

AccessFree to download

Abstract

Introduction

Multimodal learning background

SemEval-2022 Task 5 and sub-tasks

SRCB at SemEval-2022 Task 5: Pretraining Based Image to Text Late Sequential Fusion System for Multimodal Misogynous Meme Identification Yujin WangJing ZhangB, Bohua PengXudong Zhang∗ Xiaoyan Qu, Yimeng Zhuang, Song Liu Samsung Research China-Beijing (SRC-B) {[yujin1.wang](yujin1.wang) , jing97.zhang, [bohua.peng}@samsung.com](bohua.peng}@samsung.com)[ ](bohua.peng}@samsung.com){xudong.zhang, [xiaoyan11.qu}@samsung.com](xiaoyan11.qu}@samsung.com)[ ](xiaoyan11.qu}@samsung.com){ym.zhuang, [s0101.liu}@samsung.com](s0101.liu}@samsung.com) Abstract Online misogyny meme detection is an image/text multimodal classification task, the complicated relation of image and text challenges the intelligent system’s modality fusion learning capability. In this paper, we investigate the single-stream UNITER and dual-stream CLIP multimodal pretrained models on their capability to handle strong and weakly correlated image/text pairs. The XGBoost classifier with image features extracted by the CLIP model has the highest performance and being robust on domain shift. Based on this, we propose the PBR system, an ensemble system of Pretraining models, Boosting method and Rule-based adjustment, text information is fused into the system using our late sequential fusion scheme. Our system ranks 1st place on both sub-task A and sub-task B of the SemEval-2022 Task 5 Multimedia Automatic Misogyny Identification, with 0 . 834/0 .731 macro F1 scores for subtask A/B correspondingly. 1 Introduction Much of the real world’s information comes in multimodality, a combination of images, texts, audiosand so on. Multimodal understanding aims to utilize different modal of information to improve the overall system recognition intelligence or robustness (Gadzicki et al., 2020), which plays a key foundation role in cognitive AI and embodied AI. With transfer learning by large deep models and colossal corpus achieving remarkable success in vision and language domain, there is a rising interest in combining both sides’ advances to push the multimodality understanding further (Lu et al., 2019 ; Tan and Bansal, 2019 ; Chen et al., 2019 ; Li et al., 2020 ; Yu et al., 2020 ; Huo et al., 2021 ; Kim et al., 2021 ; Radford et al., 2021) . We will limit the discussion scope of multimodal to vision and language in this paper. There are two kinds of representative ontribution during Intership in Samsung Research China-Beijing. architecture of multimodal learning models, singlestream models and dual-stream models. Singlestream model fuses the image and text data at an early stage, and then feed into the model. Dualstream models design separated structure as image encoder and text encoder, and a further module is stacked on top of the unimodel encoders for crossmodal learning objectives (Tan and Bansal, 2019 ; Yu et al., 2020 ; Radford et al., 2021 ; Huo et al., 2021) . Usually per-unimodal objectives and multimodal objectives are designed to ensure that the model learns unimodal and crossmodal knowledge, like masked image prediction, masked token prediction, and text-image pairing (Chen et al., 2019 ; Kim et al., 2021) . Two kinds of data distributions are explored for the large-scale pretraining, strongly paired data (Chen et al., 2019 ; Radford et al., 2021 ; Li et al., 2020 ; Kim et al., 2021) and weakly paired data (Huo et al., 2021) . The different distributions would directly affect the correlations learned by the model, yet each pretraining corpus only falls in one pattern. The SemEval-2022 Task 5 (Fersini et al., 2022) Multimedia Automatic Misogyny Identification (MAMI) is a multimodal classification task in English. It targets the identification of misogynous memes (characterized by a pictorial content with an overlaying text a posteriori introduced by human), using the image and text from the meme as input data. It has two sub-tasks: sub-task A: 2-fold classification, to identify whether a meme is misogynous or not; sub-task B: 4-fold fine-grai

FAQs

What is the main goal of SemEval-2022 Task 5 Multimedia Automatic Misogyny Identification?

It focuses on identifying misogynous memes using both the meme image and its overlaying text, with two sub-tasks for coarse (misogynous vs. not) and fine-grained category classification.

How do the approaches differ between UNITER and CLIP in this work?

UNITER is used as a single-stream model that fuses image and text early, while CLIP is treated as a dual-stream model with separate image/text encoders for cross-modal learning.

Why is an ensemble late sequential fusion scheme used in the proposed PBR system?

Because the method aims to combine pretrained models, boosting, and rule-based adjustment while fusing text into the classification using a late sequential strategy to better handle varying image-text correlation.

Aug 26, 2026

Upload by Aria