[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-117415-en":3,"doc-seo-117415-105":30,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":4,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":27,"seo_description":14,"update_tm":28,"read_time":29},117415,7971461740909,"Levi","https://ap-avatar.wpscdn.com/davatar_155a257f0dc6eb9ab79c44ca47cae57d",8,"Research & Report","Automated Ensemble Multimodal Machine Learning for Healthcare - AutoPrognosis-M","Machine learning in medicine and healthcare has enabled many diagnostic and prognostic models, yet most existing methods rely on a single data modality and miss the broader clinical context used by clinicians. This work presents AutoPrognosis-M, a multimodal framework that automatically integrates structured tabular clinical data with medical imaging. It includes 17 imaging models and three multimodal fusion strategies, and demonstrates on a multimodal skin lesion dataset that combining fusion approaches via ensemble learning improves performance. The framework is open-sourced to accelerate clinical adoption and further innovation in multimodal healthcare ML.","arXiv :2407 . 18227v 1 [ cs .LG] 25 Jul 2024  \nAutomated Ensemble Multimodal Machine Learning for Healthcare  \nFergus Imrie 1*†, Stefan Denner2,3†, Lucas S. Brunschwig4 , Klaus Maier-Hein2,5,6 and Mihaela van der Schaar7  \n1 Department of Statistics, University of Oxford, United Kingdom.  \n2 Division of Medical Image Computing, German Cancer Research Center (DKFZ), Germany.  \n3 Medical Faculty Heidelberg, Heidelberg University, Germany.  \n4 ´Ecole Polytechnique F´ed´erale de Lausanne, Switzerland.  \n5 Pattern Analysis and Learning Group, Department of Radiation Oncology, Heidelberg University Hospital, Germany.  \n6 National Center for Tumor Diseases (NCT) Heidelberg, Germany.  \n7 Department of Applied Mathematics and Theoretical Physics, University of Cambridge, United Kingdom.  \n*Corresponding author(s). E-mail(s): [fergus.imrie@stats.ox.ac.uk](fergus.imrie@stats.ox.ac.uk);†These authors contributed equally to this work.  \nAbstract  \nThe application of machine learning in medicine and healthcare has led to the creation of numerous diagnostic and prognostic models. However, despite their success, current approaches generally issue predictions using data from a single modality. This stands in stark contrast with clinician decision-making which employs diverse information from multiple sources. While several multimodal machine learning approaches exist, significant challenges in developing multimodal systems remain that are hindering clinical adoption. In this paper, we introduce a multimodal framework, AutoPrognosis-M, that enables the integration of structured clinical (tabular) data and medical imaging using automated machine learning. AutoPrognosis-M incorporates 17 imaging models, including convolutional neural networks and vision transformers, and three distinct multimodal fusion strategies. In an illustrative application using a multimodal skin lesion dataset, we highlight the importance of multimodal machine learning and the power of combining multiple fusion strategies using ensemble learning. We have open-sourced our framework as a tool for the community and hope it will accelerate the uptake of multimodal machine learning in healthcare and spur further innovation.  \nPreprint  \n2 AutoPrognosis-Multimodal  \nIntroduction  \nMedical and healthcare data is increasingly diverse in origin and nature, encompassing patient records and imaging to genetic information and real-time biometrics. Machine learning (ML) can learn complex relationships from data to construct powerful predictive models. As a result, ML is increasingly being proposed in medicine and healthcare, particularly for diagnostic and prognostic modeling [1, 2] . However, such approaches typically make predictions based on only one type of data [3] and thus cannot incorporate all available information or consider the broader clinical context.  \nIn contrast, clinicians make decisions based on the synthesis of information from multiple sources, including imaging, structured clinical or laboratory data, and clinical notes [4] . This can be critical for accurate diagnoses and prognoses, and the absence of such information has been shown to result in lower performance and decreased clinical utility in numerous studies [5, 6] . While true across healthcare, this is perhaps particularly the case in medical imaging. For example, almost 90% of radiologists reported that additional clinical information was important and could change diagnoses compared to using the imaging alone [7] . Numerous other examples of the importance of clinical context for medical image analysis exist across specialties such as ophthalmology [8], pathology [9], and dermatology [10] .  \nMultimodal machine learning integrates multiple types and sources of data, offering a more holistic approach to model development that mirrors clinical decision-making processes. As a result, while multimodal ML remains in its infancy, models that incorporate multiple data modalities have been developed for several m","cbCaiupbhH8GN1LL","https://ap.wps.com/l/cbCaiupbhH8GN1LL","pdf",1283176,1,24,"English","en",105,"# Introduction\n## Motivation: single-modality limitations in healthcare\n## Clinician decision-making and clinical context\n## Multimodal ML and remaining adoption challenges\n## Proposed solution: AutoPrognosis-M with AutoML and ensemble learning\n# AutoPrognosis-Multimodal\n## Overview and question types enabled by multimodal learning","[{\"question\":\"Why do current healthcare ML models often underperform in real clinical use?\",\"answer\":\"They typically generate predictions using data from a single modality, which cannot incorporate the full range of clinical information and context clinicians consider.\"},{\"question\":\"What is AutoPrognosis-M and what data does it combine?\",\"answer\":\"AutoPrognosis-M is a multimodal AutoML framework that integrates structured clinical tabular data with medical imaging to build predictive models.\"},{\"question\":\"How does the framework improve multimodal modeling beyond using one fusion approach?\",\"answer\":\"It uses ensemble learning to combine multiple multimodal fusion strategies, and the paper illustrates benefits on a multimodal skin lesion dataset.\"}]","Automated Ensemble Multimodal Machine Learning for Healthcare - AutoPrognosis-M | PDF",1785675754,60,{"code":4,"msg":31,"data":32},"ok",{"site_id":24,"language":23,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":28},"automated-ensemble-multimodal-machine-learning-for-healthcare-autoprognosis-m","",{"@graph":36,"@context":85},[37,54,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,51],{"item":41,"name":42,"@type":43,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":50},"https://docshare.wps.com/document/research-report/",3,{"item":52,"name":13,"@type":43,"position":53},"https://docshare.wps.com/document/automated-ensemble-multimodal-machine-learning-for-healthcare-autoprognosis-m/117415/",4,{"url":52,"name":13,"@type":55,"author":56,"headline":13,"publisher":58,"fileFormat":61,"inLanguage":23,"description":14,"dateModified":62,"datePublished":62,"encodingFormat":61,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":57},"Person",{"url":41,"name":59,"@type":60},"DocShare","Organization","application/pdf","2026-08-02",true,{"@type":65,"interactionType":66,"userInteractionCount":4},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"Why do current healthcare ML models often underperform in real clinical use?","Question",{"text":75,"@type":76},"They typically generate predictions using data from a single modality, which cannot incorporate the full range of clinical information and context clinicians consider.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"What is AutoPrognosis-M and what data does it combine?",{"text":80,"@type":76},"AutoPrognosis-M is a multimodal AutoML framework that integrates structured clinical tabular data with medical imaging to build predictive models.",{"name":82,"@type":73,"acceptedAnswer":83},"How does the framework improve multimodal modeling beyond using one fusion approach?",{"text":84,"@type":76},"It uses ensemble learning to combine multiple multimodal fusion strategies, and the paper illustrates benefits on a multimodal skin lesion dataset.","https://schema.org",{"og:url":52,"og:type":87,"og:title":13,"og:site_name":59,"og:description":14},"article",{"robots":89,"canonical":52},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,109,114,119,122,127,130,134],{"id":20,"doc_module":4,"doc_module_name":46,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":53,"doc_module":4,"doc_module_name":46,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":46,"category_name":107,"show_sort_weight":29,"slug":108},5,"Comic","comic",{"id":110,"doc_module":4,"doc_module_name":46,"category_name":111,"show_sort_weight":112,"slug":113},6,"Technology",50,"technology",{"id":115,"doc_module":4,"doc_module_name":46,"category_name":116,"show_sort_weight":117,"slug":118},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":120,"slug":121},30,"research-report",{"id":123,"doc_module":4,"doc_module_name":46,"category_name":124,"show_sort_weight":125,"slug":126},9,"Religion & Spirituality",20,"religion-spirituality",{"id":125,"doc_module":4,"doc_module_name":46,"category_name":128,"show_sort_weight":125,"slug":129},"World Cup","world-cup",{"id":131,"doc_module":4,"doc_module_name":46,"category_name":132,"show_sort_weight":131,"slug":133},10,"Lifestyle","lifestyle",{"id":135,"doc_module":4,"doc_module_name":46,"category_name":136,"show_sort_weight":106,"slug":137},19,"General","general"]