MMRM: A Multiplex Multimodal Representation Model for Product Ranking in E-commerce Search

Jul 16, 2026

Multimodal information is crucial for e-commerce search ranking. Prior approaches fine-tune general multimodal large language models using collaborative signals and reuse the learned representations as basic item features, yet they overlook heterogeneous signals needed for multitask ranking and underutilize multimodal embeddings for user behavior modeling. The proposed MMRM unifies multiple collaborative signals via a shared backbone with task tokens and projection layers, learning multiplex item representations in one pass and deriving multiplex user representations from search-based behavior sequences. Experiments show strong efficiency and effectiveness, and the model has been deployed in JD’s search engine with significant gains for millions of daily users.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages6

Reading Time15 min

Introduction

Limitations of prior multimodal ranking methods

Heterogeneous collaborative signals and multiplex representations

Proposed MMRM framework

MMRM: A Multiplex Multimodal Representation Model for Product Ranking in E-commerce Search Zhen-Lin Chen JD.COM Beijing, China [chenzhenlin6@jd.com](chenzhenlin6@jd.com) Jianmin Chen JD.COM Beijing, China [chenjianmin.23@jd.com](chenjianmin.23@jd.com) Maosen Sheng JD.COM Beijing, China [shengmaosen.1@jd.com](shengmaosen.1@jd.com) Zhuojian Xiao JD.COM Beijing, China [xiaozhuojian5@jd.com](xiaozhuojian5@jd.com) Peng Lin JD.COM Beijing, China [linpeng47@jd.com](linpeng47@jd.com) Dongyue Wang JD.COM Beijing, China [wangdongyue@jd.com](wangdongyue@jd.com) Xiwei Zhao JD.COM Beijing, China [zhaoxiwei@jd.com](zhaoxiwei@jd.com) arXiv :2607 . 1 1030v 1 [ cs .IR] 13 Jul 2026 Abstract Multimodal information is pivotal for e-commerce search ranking. Existing works leverage multimodal data typically by fine-tuning general Multimodal Large Language Models (MLLMs) via collaborative signals, subsequently integrating the derived representations into ranking models as item features. Despite their efficacy, these methods face two primary limitations: (1) they rely on a single collaborative signal for MLLM fine-tuning, failing to exploit the heterogeneous signals essential for multitask ranking; and (2) they treat multimodal representations as regular item features in ranking models, underutilizing their latent potential for user behavior modeling. To address these challenges, we propose the Multiplex Multimodal Representation Model (MMRM), a unified framework that aligns MLLMs with diverse collaborative signals. By employing a shared backbone with task-specific tokens and projection layers, MMRM simultaneously learns from multiple signals and generates comprehensive multiplex item representations in a single inference pass. Furthermore, we introduce a multiplex user representation strategy in ranking models, which derives task-specific user representations via search-based behavior sequence modeling leveraging multiplex item representations. Extensive experiments demonstrate MMRM’s superior efficiency and effectiveness. Notably, MMRM has been successfully deployed in the JD e-commerce search engine, yielding significant performance gains for millions of daily users. CCS Concepts • Information systems → Information retrieval. This work is licensed under a Creative Commons Attribution 4 .0 International License. SIGIR ’26, Melbourne, VIC, Australia © 2026 Copyright held by the owner/author(s) . ACM ISBN 979-8-4007-2599-9/2026/07 [https://doi.org/10.1145/3805712.3808434](https://doi.org/10.1145/3805712.3808434) Keywords Multimodal Representation, Contrastive Learning, Multitask Learning, E-commerce Search System ACM Reference Format: Zhen-Lin Chen, Maosen Sheng, Peng Lin, Jianmin Chen, Zhuojian Xiao, Dongyue Wang, and Xiwei Zhao. 2026. MMRM: A Multiplex Multimodal Representation Model for Product Ranking in E-commerce Search. In Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’26), July 20–24, 2026, Melbourne, VIC, Australia. ACM, New York, NY, USA, 6 pages. [https://doi.org/10.1145/](https://doi.org/10.1145/)[ ](https://doi.org/10.1145/)3805712.3808434 1 Introduction E-commerce search engines rely on ranking models to capture finegrained user preferences across multiple objectives, such as ClickThrough Rate (CTR), Add-to-Cart Rate (ACR), and Conversion Rate (CVR) . Conventional ID-based ranking models inherently suffer from data sparsity and an inability to perceive product semantics (e.g., titles and images)[17, 25] . To address this, recent works finetune general Multimodal Large Language Models (MLLMs) using collaborative signals from e-commerce, then integrate the resulting representations into ranking models as auxiliary features[10, 11] . Despite these advancements, two limitations persist. First, existing methods rely on a single collaborative signal for MLLM finetuning, such as search-based query-to-item (q2i) o

FAQs

What are the two main limitations of existing multimodal ranking methods discussed in the abstract?

Existing methods use only a single collaborative signal to fine-tune multimodal LLMs, missing heterogeneous signals needed for multitask ranking. They also treat multimodal representations as regular item features, failing to fully exploit them for modeling user behavior.

How does MMRM align MLLMs with multiple collaborative signals?

MMRM uses a shared backbone with task-specific tokens and projection layers to simultaneously learn from multiple signals, producing multiplex item representations in a single inference pass.

How are user representations incorporated into the ranking model in MMRM?

MMRM introduces a multiplex user representation strategy that derives task-specific user representations from search-based behavior sequence modeling, using the learned multiplex item representations.

Jul 16, 2026

Upload by Genevieve