[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-126938-en":3,"doc-seo-126938-105":30,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":4,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":27,"seo_description":14,"update_tm":28,"read_time":29},126938,137451207643,"Noah","https://ap-avatar.wpscdn.com/davatar_3d24733baf745e90a7e4bdd5f77d97b2",6,"Technology","Enhancing User Behavior Modeling via Machine Learning with Combined Text and Image Data","Existing research on social media engagement modeling typically uses either image or text data, rarely integrating both modalities in a single predictive framework. This study proposes and validates a combined machine learning technique for predicting user engagement metrics using paired image-and-text inputs. Data were collected from 366,415 Facebook posts and 1,305,375 related comments. The combined model improves mean squared error by 3.5x for share count prediction and raises comment sentiment accuracy by 14% versus unimodal models.","Proceedings of the 57th Hawaii International Conference on System Sciences | 2024  \nEnhancing User Behavior Modeling via Machine Learning with Combined  \nText and Image Data  \nChad Crowe University of Nebraska at Omaha  \n[ccrowe@unomaha.edu](ccrowe@unomaha.edu)  \nDr. Brian Ricks University of Nebraska at Omaha  \n[bricks@unomaha.edu](bricks@unomaha.edu)  \nDr. Margaret Hall University of Nebraska at Omaha [mahall@unomaha.edu](mahall@unomaha.edu)  \nAbstract  \nExisting works typically operate on either image or text data from social media, but rarely work with both content types simultaneously. We propose and validate a technique for combining image and text data for predicting user engagement metrics based on social media data. We collected image and text data from 366,415 Facebook posts and a respective 1,305,375 million comments. The combined model achieves a 3 .5x improvement in mean squared error when predicting share count and a 14% improvement for comment sentiment over single data type models. Finally, the study demonstrates the ability to pick more performant advertisement out of 16.7 billion pairs; the resulting machine learning models successfully predicts for a greater comment sentiment, comment count, and share count 93%, 65%, and 63% of the time.  \nKeywords: social media, machine learning, deep neural networks, ensemble models, advertisements  \n1. Introduction  \nTwo concurrent challenges exist for researchers and practitioners in using social media data to model behaviors (Miranda et al., 2022; Padmanabhan et al., 2022) . The first leans heavily into theoretical aspects of (digital) behaviors and centers on questions like why, when, and under what circumstances users engage in specific behaviors; how to elicit ordisincentivize behaviors; or understanding the spectrum between short-term and long-term impacts of behavior (Miranda et al., 2022) . The other challenge is more methodological in nature. Scholars addressing this second set of challenges seek to develop and deploy ever-more accurate, yet simplified, models that can work  \nacross platforms or data types to support research and industry (Padmanabhan et al., 2022) .  \nWhereas theoretical approaches to understanding digital user engagement are quite mature (See Section 2.1, Related Work), in some ways, use case-agnostic methods and architectures are lagging behind. Nowhere is this clearer than in the use of multiple data types in machine learning (ML) models. Existing research has established that image media, text media, and social media post sentiment individually affect member behaviors (Gelli et al., 2015; Khosla et al., 2014; Liet al., 2015; Segalin et al., 2017; Straton et al., 2017) . However, data types (i.e., text and images) are rarely analyzed simultaneously. For example, a user might like a post because its description of the image is funny. Therefore, it is reasonable to assume the interaction of text and images affects user engagement. We posit that a significant overlap interaction exists between each media format (Sabate et al., 2014) .  \nThe gap exists because researchers rarely incorporate multiple data types, despite the availability of both image and text data (See Section 2.2, Related Work) . This gap is driven in part by the maturity of image processing methodologies and corresponding technological requirements. Image processing is an emergent application, which still requires considerable computing power and technological fluency. Many prior works of note only focus on a single data type, we assume, because working with a single data type is simpler and often performs quite well for its purpose, including predicting gender classification (Hassner & Tal, 2015), detecting sarcasm (Poria et al., 2016), profiling (Segalin et al., 2017), and predicting social media popularity (Gelli et al., 2015) . Some attempts exist that link the two data types, i.e., performing sentiment analysis of posts with images (Y. Wang et al.,  \nURI: [https://hdl.handl","cbCaieLlSD1VEduA","https://ap.wps.com/l/cbCaieLlSD1VEduA","pdf",1930198,1,10,"English","en",105,"# Introduction\n## Research questions and scope\n# Related Work\n## Social media modeling with single vs combined data types","[{\"question\":\"Why is combining text and image data important for user behavior modeling?\",\"answer\":\"Because prior work often models image or text separately, while user engagement can be driven by interactions between what a post shows and what its text describes.\"},{\"question\":\"What data sources and engagement metrics are used in the study?\",\"answer\":\"The study uses image and text data from Facebook posts and associated comments, predicting metrics such as share count, comment sentiment, and comment count.\"},{\"question\":\"How does the combined model perform compared with single-data-type models?\",\"answer\":\"It improves mean squared error for share count by 3.5x and improves comment sentiment by 14% compared with models trained on only one data type.\"}]","Enhancing User Behavior Modeling via Machine Learning with Combined Text and Image Data | PDF",1785935784,25,{"code":4,"msg":31,"data":32},"ok",{"site_id":24,"language":23,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":28},"enhancing-user-behavior-modeling-via-machine-learning-with-combined-text-and-image-data","",{"@graph":36,"@context":85},[37,54,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,51],{"item":41,"name":42,"@type":43,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":50},"https://docshare.wps.com/document/technology/",3,{"item":52,"name":13,"@type":43,"position":53},"https://docshare.wps.com/document/enhancing-user-behavior-modeling-via-machine-learning-with-combined-text-and-image-data/126938/",4,{"url":52,"name":13,"@type":55,"author":56,"headline":13,"publisher":58,"fileFormat":61,"inLanguage":23,"description":14,"dateModified":62,"datePublished":62,"encodingFormat":61,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":57},"Person",{"url":41,"name":59,"@type":60},"DocShare","Organization","application/pdf","2026-08-05",true,{"@type":65,"interactionType":66,"userInteractionCount":4},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"Why is combining text and image data important for user behavior modeling?","Question",{"text":75,"@type":76},"Because prior work often models image or text separately, while user engagement can be driven by interactions between what a post shows and what its text describes.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"What data sources and engagement metrics are used in the study?",{"text":80,"@type":76},"The study uses image and text data from Facebook posts and associated comments, predicting metrics such as share count, comment sentiment, and comment count.",{"name":82,"@type":73,"acceptedAnswer":83},"How does the combined model perform compared with single-data-type models?",{"text":84,"@type":76},"It improves mean squared error for share count by 3.5x and improves comment sentiment by 14% compared with models trained on only one data type.","https://schema.org",{"og:url":52,"og:type":87,"og:title":13,"og:site_name":59,"og:description":14},"article",{"robots":89,"canonical":52},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,113,118,123,128,131,134],{"id":20,"doc_module":4,"doc_module_name":46,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":53,"doc_module":4,"doc_module_name":46,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":46,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":111,"slug":112},50,"technology",{"id":114,"doc_module":4,"doc_module_name":46,"category_name":115,"show_sort_weight":116,"slug":117},7,"Healthcare",40,"healthcare",{"id":119,"doc_module":4,"doc_module_name":46,"category_name":120,"show_sort_weight":121,"slug":122},8,"Research & Report",30,"research-report",{"id":124,"doc_module":4,"doc_module_name":46,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":46,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":21,"doc_module":4,"doc_module_name":46,"category_name":132,"show_sort_weight":21,"slug":133},"Lifestyle","lifestyle",{"id":135,"doc_module":4,"doc_module_name":46,"category_name":136,"show_sort_weight":106,"slug":137},19,"General","general"]