[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-82217-en":3,"doc-seo-82217-105":29,"detail-sidebar-cat-0-en-105":90},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":4,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},82217,1374391974468,"Eden","https://ap-avatar.wpscdn.com/davatar_29158cc5080c5b710cf443261637dec0",7,"Healthcare","MedRealMM: A Real-World Multimodal Benchmark for Chinese Online Medical Consultation","Large language models are increasingly used in online medical consultation, yet existing benchmarks do not match real clinical workflows. Many rely on synthetic dialogues, omit patient-uploaded medical images, and assess open-ended answers with multiple-choice or lexical-overlap metrics that fail to reflect clinical quality. MedRealMM introduces a large-scale multimodal benchmark built from de-identified patient-doctor interactions from a nationwide Chinese internet hospital. It extracts clinically challenging moments, generates standardized next-response tasks with physician-refined rubrics, and evaluates both text-only and multimodal LLMs using 5,620 real cases across 64 departments.","arXiv :2607 .09 142v 1 [ cs .AI] 10 Jul 2026  \nMEDREALMM: A REAL-WORLD MULTIMODAL BENCHMARK FOR CHINESE ONLINE MEDICAL CONSULTATION  \nRunhan Shi2 ,†,∗ Quan Zhou3 Yuqian Xu4 Shuai Yang 1 Xin Wu 1 Zitong Zhou5 ,† Hui Liu 1  \nBin Cha 1 Zheming Wang 1 Liya Li 1 Wei Wei 1 Haoyuan Hu 1 Jun Xu 1  \n1JD Health International Inc.  \n2 Shanghai Jiao Tong University  \n3National University of Singapore  \n4University of North Carolina Chapel Hill  \n5University of Pennsylvania  \nABSTRACT  \nLarge language models (LLMs) are increasingly deployed in online medical consultation, yet existing benchmarks remain poorly aligned with real clinical practice. Many rely on synthetic conversations or patient simulators, omit patient-uploaded medical images, or evaluate open-ended clinical responses using multiple-choice or lexical-overlap metrics that poorly reflect clinical quality. We introduce MedRealMM, a large-scale benchmark for multimodal online medical consultation built from de-identified patient-doctor interactions collected from a nationwide Chinese internet hospital.  \nMedRealMM uses a Multimodal Clinical Challenge Point (MCCP) extraction framework to identify clinically demanding moments in authentic consultation trajectories and converts each into a standardized next-response generation task while preserving the preceding text-image context. Each instance is paired with a case-specific rubric refined by physicians that rewards clinically desirable behaviorsand penalizes unsafe, unsupported, or contradictory responses. The current release contains 5,620 real-world multimodal cases spanning 64 clinical departments. We evaluate 19 general-purpose and medical-specialized LLMs, including text-only and multimodal systems. Our results show that image information is critical for reliable clinical performance and that current frontier models remain below the online physician response. Although some frontier models satisfy as many or more positive clinical criteria than physicians, they trigger more negative criteria, indicating that safety-sensitive error avoidance remains a central bottleneck. MedRealMM offers a realistic and reproducible benchmark for evaluating multimodal medical reasoning in real-world online consultation. The dataset will be publicly available on Hugging Face at [https://huggingface.co/datasets/jdh-algo/MedRealMM](https://huggingface.co/datasets/jdh-algo/MedRealMM).  \n1 Introduction  \nOnline medical consultation has become an important component of China’s digital health system. Patients increasingly interact with physicians through asynchronous, platform-mediated encounters that combine free-text messages, structured medical records, and patient-uploaded images. This trend is driven by the uneven distribution of offline medical  \n†Work done during the internship at JD Health International Inc.*  \nCorresponding author  \nFigure 1: Three gaps between existing medical LLM benchmarks and real-world online consultation: (a) unrealistic consultation trajectories,(b) missing multimodal evidence, and (c) inadequate open-ended response evaluation.  \nresources, the growing population of people with chronic diseases, and national digital health initiatives [1, 2] . For example, JD Health ([https://ir.jdhealth.com](https://ir.jdhealth.com)), one of the largest online healthcare platforms in China, reported 217.7 million annual active users at the end of 2025 . Its Internet Hospital handled more than 180 million consultationsin 2024, averaging more than 490,000 consultations per day [3, 4] . The scale of these services has created a growing demand for AI systems that can assist physicians during online consultation.  \nRecent advances in large language models (LLMs) have made them promising assistants for online medical consultation and have been increasingly explored for applications such as medical question answering, consultation assistance, and clinical decision support [5, 6, 7] . However, whether these models can reliably support real-w","cbCaivyzYbSjju8r","https://ap.wps.com/l/cbCaivyzYbSjju8r","pdf",6473186,1,25,"English","en",105,"# Introduction\n## Gaps in existing medical LLM benchmarks\n## MedRealMM benchmark construction\n## Evaluation setup and findings","[{\"question\":\"What problem does MedRealMM address in current medical LLM benchmarks?\",\"answer\":\"Existing benchmarks are poorly aligned with real clinical practice, often using synthetic conversations, excluding patient-uploaded medical images, and evaluating open-ended responses with metrics that do not capture clinical appropriateness and safety.\"},{\"question\":\"How is MedRealMM constructed from real online consultation data?\",\"answer\":\"MedRealMM uses the Multimodal Clinical Challenge Point (MCCP) extraction framework to identify clinically demanding moments in authentic consultation trajectories, then converts each moment into a standardized next-response generation task while preserving the prior text-image context.\"},{\"question\":\"What evaluation results does the paper report about image information and model safety?\",\"answer\":\"The results show image information is critical for reliable clinical performance and that frontier models remain below online physicians; some models meet more positive criteria yet trigger more negative criteria, highlighting safety-sensitive error avoidance as a bottleneck.\"}]",1784178911,63,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":85,"head_meta":87,"extra_data":89,"updated_unix":27},"medrealmm-a-real-world-multimodal-benchmark-for-chinese-online-medical-consultation","",{"@graph":35,"@context":84},[36,53,67],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/healthcare/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/medrealmm-a-real-world-multimodal-benchmark-for-chinese-online-medical-consultation/82217/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":61,"encodingFormat":60,"isAccessibleForFree":62,"interactionStatistic":63},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-16",true,{"@type":64,"interactionType":65,"userInteractionCount":4},"InteractionCounter",{"@type":66},"ViewAction",{"@type":68,"mainEntity":69},"FAQPage",[70,76,80],{"name":71,"@type":72,"acceptedAnswer":73},"What problem does MedRealMM address in current medical LLM benchmarks?","Question",{"text":74,"@type":75},"Existing benchmarks are poorly aligned with real clinical practice, often using synthetic conversations, excluding patient-uploaded medical images, and evaluating open-ended responses with metrics that do not capture clinical appropriateness and safety.","Answer",{"name":77,"@type":72,"acceptedAnswer":78},"How is MedRealMM constructed from real online consultation data?",{"text":79,"@type":75},"MedRealMM uses the Multimodal Clinical Challenge Point (MCCP) extraction framework to identify clinically demanding moments in authentic consultation trajectories, then converts each moment into a standardized next-response generation task while preserving the prior text-image context.",{"name":81,"@type":72,"acceptedAnswer":82},"What evaluation results does the paper report about image information and model safety?",{"text":83,"@type":75},"The results show image information is critical for reliable clinical performance and that frontier models remain below online physicians; some models meet more positive criteria yet trigger more negative criteria, highlighting safety-sensitive error avoidance as a bottleneck.","https://schema.org",{"og:url":51,"og:type":86,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":88,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":91},[92,96,100,104,109,114,117,122,127,130,134],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":93,"show_sort_weight":94,"slug":95},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":97,"show_sort_weight":98,"slug":99},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":101,"show_sort_weight":102,"slug":103},"Exam",70,"exam",{"id":105,"doc_module":4,"doc_module_name":45,"category_name":106,"show_sort_weight":107,"slug":108},5,"Comic",60,"comic",{"id":110,"doc_module":4,"doc_module_name":45,"category_name":111,"show_sort_weight":112,"slug":113},6,"Technology",50,"technology",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":115,"slug":116},40,"healthcare",{"id":118,"doc_module":4,"doc_module_name":45,"category_name":119,"show_sort_weight":120,"slug":121},8,"Research & Report",30,"research-report",{"id":123,"doc_module":4,"doc_module_name":45,"category_name":124,"show_sort_weight":125,"slug":126},9,"Religion & Spirituality",20,"religion-spirituality",{"id":125,"doc_module":4,"doc_module_name":45,"category_name":128,"show_sort_weight":125,"slug":129},"World Cup","world-cup",{"id":131,"doc_module":4,"doc_module_name":45,"category_name":132,"show_sort_weight":131,"slug":133},10,"Lifestyle","lifestyle",{"id":135,"doc_module":4,"doc_module_name":45,"category_name":136,"show_sort_weight":105,"slug":137},19,"General","general"]