[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-81494-en":3,"doc-seo-81494-105":30,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":21,"is_downloadable":21,"audit_status":21,"page_count":22,"language":23,"language_code":24,"site_id":25,"html_lang":24,"table_of_contents":26,"faqs":27,"seo_title":13,"seo_description":14,"update_tm":28,"read_time":29},81494,1099513958607,"Jiven","https://ap-avatar.wpscdn.com/avatar/100002390cf8733938c?x-image-process=image/resize,m_fixed,w_180,h_180&k=1778829742770036399",8,"Research & Report","Contrastive Learning on Multimodal Analysis of Electronic Health Records","Electronic health record (EHR) systems capture multimodal clinical data, including structured codes and unstructured clinical notes, but existing research often treats modalities in isolation or merges them with simplistic strategies that miss their intrinsic synergy. A multimodal feature embedding generative model and a multimodal contrastive loss are proposed to learn EHR representations. Theoretical analysis shows multimodal learning’s advantage over single-modality learning and links the loss to SVD of a pointwise mutual information matrix, yielding a privacy-preserving algorithm. Simulation results and validation on real-world EHR data support clinical usefulness.","arXiv :2403 . 14926v3 [ stat .ML] 10 Jul 2026  \nContrastive Learning on Multimodal Analysis of Electronic Health Records  \nTianxi Cai 1 ,2⋆, Feiqing Huang 1⋆, Ryumei Nakada2 ,3⋆ ,  \nLinjun Zhang3⋆, Doudou Zhou4⋆  \n1 Department of Biostatistics, Harvard T.H. Chan School of Public Health, Boston, MA  \n2 Department of Biomedical Informatics, Harvard Medical School, Boston, MA  \n3 Department of Statistics, Rutgers University, Piscataway, NJ  \n4 Department of Statistics and Data Science, National University of Singapore, Singapore  \n⋆ alphabetical order  \nAbstract  \nElectronic health record (EHR) systems capture a wealth of multimodal clinical data, encompassing both structured clinical codes and unstructured clinical notes. Yet, many EHR-focused studies have traditionally examined these modalities in isolation or combined them using simplistic methods, overlooking the intrinsic synergy between them. In reality, these modalities are deeply interconnected, each containing clinically relevant and complementary information that, when integrated effectively, can provide a more comprehensive understanding of patient health. Despite the success of multimodal contrastive learning in vision-language applications, its potential remains under-explored in multimodal EHR, particularly in terms of theoretical understanding. To support statistical analysis of multimodal EHR data, we propose amultimodal feature embedding generative model and design a multimodal contrastive loss to learn EHR feature representations. Our theoretical analysis demonstrates the effectiveness of multimodal learning over single-modality learning and connects the solution of the loss function to the singular value decomposition of a pointwise mutual information matrix. This connection leads to a privacy-preserving algorithm tailored for multimodal EHR representation learning. Simulation studies show that the proposed algorithm performs well under a variety of configurations. We further validate its clinical utility using real-world EHR data.  \nKeywords: Natural language processing, textual data, structured data, representation learning, singular value decomposition.  \n1 Introduction  \nThe growing accessibility of Electronic Health Record (EHR) data presents numerous opportunities for clinical research, ranging from patient profiling to predicting medical events. However, the complexity increases with the multimodal nature of EHR data, which encompasses diverse data from patient demographics and genetic information to unstructured textual data like clinical notes, as well as structured data such as diagnostic and procedure codes, medication orders, and lab results.  \nA key challenge in EHR-focused research lies in effectively merging these different data types and ensuring that their clinical aspects are meaningfully and coherently represented. Research has shown the benefits of integrating structured and unstructured data for tasks like automated clinical code assignment (Scheurwegs et al., 2016), managing chronic diseases (Sheikhalishahi et al., 2019), and pharmacovigilance (Stang et al., 2010) . While these different modalities serve as complementary data sources, there is significant overlap and correlation among these modalities (Qiao et al., 2019) . Joint representation of both structured and narrative data into a more manageable low-dimensional space where similar features are grouped closely can significantly improve the utility of both data types. This representation learning technique has gained popularity for its ability to capture and represent the intricate relationships among various EHR features.  \nDespite extensive research on EHR feature representation, most studies have focused on datasets with a single modality, including structured (Choi et al., 2016a; Kartchner et al. , 2017; Hong et al., 2021; Zhou et al., 2022) and unstructured data (De Vine et al., 2014; Choi et al., 2016b; Beam et al., 2019; Alsentzer et al., 2019; Huang et al., 2020; Lehman and J","cbCaitdNehtduYEu","https://ap.wps.com/l/cbCaitdNehtduYEu","pdf",4419756,4,1,86,"English","en",105,"# Introduction\n## Multimodal EHR and key challenges\n## Related unimodal and multimodal representation learning","[{\"question\":\"为什么需要将EHR的结构化数据与非结构化文本进行多模态对齐与融合？\",\"answer\":\"结构化代码与临床笔记包含互补的临床信息且存在重叠与相关性。有效融合能把不同模态映射到可管理的低维空间，使相似特征更紧密，从而提升两类数据的整体效用。\"},{\"question\":\"文档提出的核心方法是什么，用于学习多模态EHR表示？\",\"answer\":\"提出一种多模态特征嵌入生成模型，并设计多模态对比损失来学习EHR特征表示。该框架旨在同时利用结构化与非结构化数据的协同信息。\"},{\"question\":\"理论分析如何解释多模态对比学习的有效性，并与隐私保护算法产生关联？\",\"answer\":\"理论分析证明多模态学习优于单模态学习，并将损失函数的解与点互信息矩阵的奇异值分解（SVD）联系起来。该联系进一步导出面向多模态EHR表示学习的隐私保护算法。\"}]",1784173800,217,{"code":4,"msg":31,"data":32},"ok",{"site_id":25,"language":24,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":28},"contrastive-learning-on-multimodal-analysis-of-electronic-health-records","",{"@graph":36,"@context":85},[37,53,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,51],{"item":41,"name":42,"@type":43,"position":21},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":50},"https://docshare.wps.com/document/research-report/",3,{"item":52,"name":13,"@type":43,"position":20},"https://docshare.wps.com/document/contrastive-learning-on-multimodal-analysis-of-electronic-health-records/81494/",{"url":52,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":24,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":41,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-23","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"为什么需要将EHR的结构化数据与非结构化文本进行多模态对齐与融合？","Question",{"text":75,"@type":76},"结构化代码与临床笔记包含互补的临床信息且存在重叠与相关性。有效融合能把不同模态映射到可管理的低维空间，使相似特征更紧密，从而提升两类数据的整体效用。","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"文档提出的核心方法是什么，用于学习多模态EHR表示？",{"text":80,"@type":76},"提出一种多模态特征嵌入生成模型，并设计多模态对比损失来学习EHR特征表示。该框架旨在同时利用结构化与非结构化数据的协同信息。",{"name":82,"@type":73,"acceptedAnswer":83},"理论分析如何解释多模态对比学习的有效性，并与隐私保护算法产生关联？",{"text":84,"@type":76},"理论分析证明多模态学习优于单模态学习，并将损失函数的解与点互信息矩阵的奇异值分解（SVD）联系起来。该联系进一步导出面向多模态EHR表示学习的隐私保护算法。","https://schema.org",{"og:url":52,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":52},"index,follow",{"doc_id":7,"site_id":25},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":21,"doc_module":4,"doc_module_name":46,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":20,"doc_module":4,"doc_module_name":46,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":46,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":46,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":46,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":46,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":46,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":46,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":46,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]