M6: Multi-Modality-to-Multi-Modality Multitask Mega-transformer for Unified Pretraining

Aug 25, 2026

Multimodal pretraining has proven effective for cross-modal representation learning, yet existing efforts are constrained mainly to English data and lack large-scale Chinese resources. This work introduces the largest Chinese pretraining dataset, M6-Corpus, with over 1.9× images and 292× texts spanning domains such as encyclopedias, question answering, and forum discussions. A unified model, M6, performs multi-modal-to-text and transfer tasks via multitask mega-transformer pretraining. Scaling to 10B parameters achieves strong gains on downstream tasks and shows strong zero-shot potential.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages11

Reading Time28 min

AccessFree to download

Abstract

Introduction

Challenges in Multimodal Pretraining for Chinese

M6-Corpus: Large-Scale Chinese Multimodal Dataset

M6: Unified Multi-Modal Mega-transformer

M6: Multi-Modality-to-Multi-Modality Multitask Mega-transformer for Unified Pretraining Junyang Lin 1∗, Rui Men 1∗, An Yang 1∗, Chang Zhou 1 , Yichang Zhang 1 , Peng Wang 1 , Jingren Zhou 1 , Jie Tang2†, Hongxia Yang 1† 1DAMO Academy, Alibaba Group 2Tsinghua University {junyang.ljy,[menrui.mr](menrui.mr), ya235025,ericzhou.zc,yichang.zyc,zheluo.wp,jingren.zhou,[yang.yhx}@alibaba-inc.com](yang.yhx}@alibaba-inc.com) [jietang@tsinghua.edu.cn](jietang@tsinghua.edu.cn) ABSTRACT Multimodal pretraining has demonstrated success in the downstream tasks of cross-modal representation learning. However, it is limited to the English data, and there is still a lack of large-scale dataset for multimodal pretraining in Chinese. In this work, we propose the largest dataset for pretraining in Chinese, which consists of over 1. 9􀀩 􀀗 images and 292􀀜􀀗 texts. The dataset has large coverage over domains, including encyclopedia, question answering, forum discussion, etc. Besides, we propose a method called M6, referring to Multi-Modality-to-Multi-Modality Multitask Mega-transformer, for unified pretraining on the data of single modality and multiple modalities. The model is pretrained with our proposed tasks, including text-to-text transfer, image-to-text transfer, as well as multimodality-to-text transfer. The tasks endow the model with strong capability of understanding and generation. We scale the model to 10 billion parameters, and build the largest pretrained model in Chinese. Experimental results show that our proposed M6 outperforms the baseline in a number of downstream tasks concerning both single modality and multiple modalities, and the 10􀀗 -parameter pretrained model demonstrates strong potential in the setting of zero-shot learning. CCS CONCEPTS • Computing methodologies → Natural language processing; Computer vision. KEYWORDS Multi-modal pretraining; Large-scale pretraining; Cross-modal understanding and generation ACM Reference Format: Junyang Lin1∗ , Rui Men1∗ , An Yang1∗ , Chang Zhou1 , Yichang Zhang1 , Peng Wang1 ,, Jingren Zhou1 , Jie Tang2†, Hongxia Yang1†. 2021. M6: MultiModality-to-Multi-Modality Multitask Mega-transformer for Unified Pretraining. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD’21), August 14–18, 2021, Virtual Event, Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission [and/or a fee. Request permissions from permissions@acm.org](and/or a fee. Request permissions from permissions@acm.org). KDD’21, August 14–18, 2021, Virtual Event, Singapore © 2021 Association for Computing Machinery. ACM ISBN 978-1-4503-8332-5/21/08…$15.00 [https://doi.org/10.1145/3447548.3467206](https://doi.org/10.1145/3447548.3467206) Singapore. ACM, New York, NY, USA, 11 pages. [https://doi.org/10.1145/](https://doi.org/10.1145/)[ ](https://doi.org/10.1145/)3447548.3467206 1 INTRODUCTION Pretraining has recently greatly promoted the development of natural language processing (NLP) . A series of studies in pretraining [1, 2, 8, 15, 17, 18, 24, 29, 35, 41, 46] have gradually pushed the limit of model performance in natural language understanding and even natural language generation. Besides, pretraining leveraging large-scale data enables the building of extremely large models with large capacity. The recent GPT-3 with over 175 billion parameters demonstrates that large models can achieve outstanding performance even in the setting of few-shot or zero-shot learning. The rapid development of pretraining in NL

FAQs

What is the main contribution of this work for Chinese multimodal pretraining?

It proposes M6-Corpus, a large-scale Chinese dataset for multimodal pretraining, along with a unified model (M6) for multimodal understanding and generation tasks.

How does M6-Corpus cover data across domains?

It is collected from webpages and includes multiple types of content, covering areas such as encyclopedias, question answering, forum discussions, and product descriptions.

What tasks does the M6 model use during pretraining?

The model is pretrained with tasks that support text-to-text transfer, image-to-text transfer, and multimodality-to-text transfer, enabling both single-modal and multimodal capabilities.

Aug 25, 2026

Upload by Ethan Miller