[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-seo-578972-105":3,"detail-sidebar-cat-0-en-105":81,"doc-detail-578972-en":130},{"code":4,"msg":5,"data":6},0,"ok",{"site_id":7,"language":8,"slug":9,"title":10,"keywords":11,"description":12,"schema_data":13,"social_meta":74,"head_meta":76,"extra_data":78,"updated_unix":80},105,"en","multimodal-engagement-and-sentiment-analytics-in-health-science-education-a-learning-analytics-framework-integrating-ai-and-pedagogical-theory","Multimodal Engagement and Sentiment Analytics in Health Science Education - A Learning Analytics Framework Integrating AI and Pedagogical Theory","","Online learning environments often lack the social and pedagogical cues of face-to-face classrooms, making real-time evaluation of engagement and emotional states difficult. This study proposes a multimodal learning analytics framework that integrates visual and textual signals to infer learner emotions and engagement, improving interpretability and pedagogical usefulness. It builds two datasets for facial-expression emotion (10,000 grayscale images) and engagement indicators (4,000 images) plus 1,667 MOOC feedback responses for sentiment analysis. CNNs classify emotion and engagement, while a fine-tuned BERT model performs sentiment detection. A rule-based integration engine generates multidimensional behavioral typologies; results show >92% validation accuracy for CNN tasks and F1=0.87/88.1% accuracy for sentiment.",{"@graph":14,"@context":73},[15,34,56],{"@type":16,"itemListElement":17},"BreadcrumbList",[18,23,27,31],{"item":19,"name":20,"@type":21,"position":22},"https://docshare.wps.com","Home","ListItem",1,{"item":24,"name":25,"@type":21,"position":26},"https://docshare.wps.com/document/","Document",2,{"item":28,"name":29,"@type":21,"position":30},"https://docshare.wps.com/document/research-report/","Research & Report",3,{"item":32,"name":10,"@type":21,"position":33},"https://docshare.wps.com/document/multimodal-engagement-and-sentiment-analytics-in-health-science-education-a-learning-analytics-framework-integrating-ai-and-pedagogical-theory/578972/",4,{"url":32,"name":10,"@type":35,"image":36,"author":41,"headline":10,"publisher":44,"fileFormat":47,"inLanguage":8,"description":12,"dateModified":48,"datePublished":49,"encodingFormat":47,"isAccessibleForFree":50,"interactionStatistic":51},"DigitalDocument",{"url":37,"@type":38,"width":39,"height":40},"https://docshare.wps.com/thumbnails/multimodal-engagement-and-sentiment-analytics-in-health-science-education-a-learning-analytics-framework-integrating-ai-and-pedagogical-theory/578972.png","ImageObject",300,407,{"name":42,"@type":43},"Noah","Person",{"url":19,"name":45,"@type":46},"DocShare","Organization","application/pdf","2026-11-15","2026-10-15",true,{"@type":52,"interactionType":53,"userInteractionCount":55},"InteractionCounter",{"@type":54},"ViewAction",7,{"@type":57,"mainEntity":58},"FAQPage",[59,65,69],{"name":60,"@type":61,"acceptedAnswer":62},"Why are current engagement and emotion evaluation methods insufficient in online learning?","Question",{"text":63,"@type":64},"They often rely on single-modality signals such as facial expression recognition or text sentiment analysis, which can oversimplify behavior and miss how cognitive and emotional engagement interact.","Answer",{"name":66,"@type":61,"acceptedAnswer":67},"What multimodal data sources does the proposed framework use?",{"text":68,"@type":64},"It combines visual information (facial expression images and engagement-indicator images) with textual feedback from massive open online courses for sentiment analysis.",{"name":70,"@type":61,"acceptedAnswer":71},"Which models are used for emotion, engagement, and sentiment detection?",{"text":72,"@type":64},"Convolutional neural networks (CNNs) are used for emotion and engagement classification, and a fine-tuned BERT model is used for sentiment analysis.","https://schema.org",{"og:url":32,"og:type":75,"og:title":10,"og:site_name":45,"og:description":12},"article",{"robots":77,"canonical":32},"index,follow",{"doc_id":79,"site_id":7},578972,1792189489,{"code":4,"msg":82,"data":83},"success",[84,88,92,96,101,106,110,114,119,122,126],{"id":22,"doc_module":4,"doc_module_name":25,"category_name":85,"show_sort_weight":86,"slug":87},"Story & Novel",90,"story-novel",{"id":26,"doc_module":4,"doc_module_name":25,"category_name":89,"show_sort_weight":90,"slug":91},"Literature",80,"literature",{"id":33,"doc_module":4,"doc_module_name":25,"category_name":93,"show_sort_weight":94,"slug":95},"Exam",70,"exam",{"id":97,"doc_module":4,"doc_module_name":25,"category_name":98,"show_sort_weight":99,"slug":100},5,"Comic",60,"comic",{"id":102,"doc_module":4,"doc_module_name":25,"category_name":103,"show_sort_weight":104,"slug":105},6,"Technology",50,"technology",{"id":55,"doc_module":4,"doc_module_name":25,"category_name":107,"show_sort_weight":108,"slug":109},"Healthcare",40,"healthcare",{"id":111,"doc_module":4,"doc_module_name":25,"category_name":29,"show_sort_weight":112,"slug":113},8,30,"research-report",{"id":115,"doc_module":4,"doc_module_name":25,"category_name":116,"show_sort_weight":117,"slug":118},9,"Religion & Spirituality",20,"religion-spirituality",{"id":117,"doc_module":4,"doc_module_name":25,"category_name":120,"show_sort_weight":117,"slug":121},"World Cup","world-cup",{"id":123,"doc_module":4,"doc_module_name":25,"category_name":124,"show_sort_weight":123,"slug":125},10,"Lifestyle","lifestyle",{"id":127,"doc_module":4,"doc_module_name":25,"category_name":128,"show_sort_weight":97,"slug":129},19,"General","general",{"code":4,"msg":82,"data":131},{"doc_id":79,"user_id":132,"nickname":42,"user_avatar":133,"doc_module":4,"category_id":111,"category_name":29,"doc_title":10,"doc_description":12,"doc_content":134,"file_id":135,"file_url":136,"file_type":137,"file_size":138,"view_count":55,"is_deleted":4,"is_public":22,"is_downloadable":22,"audit_status":22,"page_count":139,"language":140,"language_code":8,"site_id":7,"html_lang":8,"table_of_contents":141,"faqs":142,"seo_title":143,"seo_description":12,"update_tm":144,"read_time":145,"preview_page_count":4},8796095462418,"https://ap-avatar.wpscdn.com/avatar/80000253c1241d02b47?x-image-process=image/resize,m_fixed,w_180,h_180&k=1778826106357471780","International Review of Research in Open and Distributed Learning Volume 27, Number 1  \nFebruary – 2026  \nMultimodal Engagement and Sentiment Analytics in Health Science Education: A Learning Analytics Framework Integrating AI and Pedagogical Theory  \nHao Fang1 , Aiwei Mu2, Guosheng Xing3, Xingyu Chen4, and Seng Yue Wong5*  \n1 Institute for Advanced Studies, University of Malaya, Kuala Lumpur, Malaysia; 2International College, Krirk University, Bangkok, Thailand; 3Graduate School, SuanSunandha Rajabhat University, Bangkok, Thailand; 4Faculty of Education, University of Malaya, Kuala Lumpur, Malaysia; 5Academy of Malay Studies, University of Malaya, Kuala Lumpur, Malaysia; *Corresponding Author  \nAbstract  \nOnline learning environments tend not to provide the social and pedagogical cues of physical classrooms, so evaluating student engagement and emotional states in real time becomes challenging. Current methods depend mainly upon facial expression recognition or textual sentiment analysis, constraining the depth and accuracy of behavioral interpretation. This research suggests a multimodal learning analytics framework that combines visual and textual data to infer learner emotions and engagement for improving the interpretability, responsiveness, and pedagogical value of learning analytics systems in digital education. Two datasets were created: (a) a facial expression dataset of 10,000 grayscale images annotated over five emotion categories and (b) an engagement dataset of 4,000 images annotated according to behavioral indicators. Concurrently, 1,667 learner feedback responses from massive open online courses were prepared for sentiment analysis. Convolutional neural networks (CNNs) were used for emotion and engagement classification, and a fine-tuned BERT (bidirectional encoder representations from transformers) model for sentiment analysis. A rule-based integration engine combined outputs to create multidimensional behavioural typologies. The CNN models reached >92% validation accuracy for both emotion detection and engagement detection tasks, whereas the BERT sentiment classifier achieved F1 = 0.87 and 88.1% accuracy. The multimodal integration procedure identified four unique learner behavior typologies (e.g., students who were cognitively engaged but visually disengaged) . The framework offers an accurate, interpretable, and scalable real-time learning analytics solution. Compared with previous methods, it overcomes significant limitations and offers a useful resource for facilitating adaptive, data-based instruction interventions, especially in online and health science education.  \nKeywords: health science education, learning analytics, sentiment analysis, emotion detection, BERT, engagement typology, cognitive presence, multimodal AI  \nIntroduction  \nThe quick growth of virtual learning environments, especially during the post-pandemic period, has created the need for more conscious monitoring of and improvement mechanisms for learner motivation and emotional well-being. Although electronic learning environments are scalable and accessible, they may be devoid of the social cues present in conventional classrooms that enable teachers to monitor learner motivation, engagement, and emotional state in real time (Selim et al., 2022) . The lack of such indicators has driven the development of affective computing in learning to decode learners’ emotional and cognitive states based on machine learning and artificial intelligence (AI) . Traditional research in this area has used facial expression analysis or text-based feedback to make inferences about students’ emotional states and engagement levels (Gambo et al., 2022; Selim et al., 2022) . However, these unimodal approaches often oversimplify human behavior and fail to capture the interplay between cognitive and emotional engagement. Health science education, in particular, requires both cognitive depth and emotional resilience. As online delivery expands access, it simulta","cbCaidoA1WT9DFjk","https://ap.wps.com/l/cbCaidoA1WT9DFjk","pdf",1178521,25,"English","# Abstract\n# Introduction\n## Background and motivation\n## Multimodal analytics approach and pedagogical grounding\n## Related work and limitations","[{\"question\":\"Why are current engagement and emotion evaluation methods insufficient in online learning?\",\"answer\":\"They often rely on single-modality signals such as facial expression recognition or text sentiment analysis, which can oversimplify behavior and miss how cognitive and emotional engagement interact.\"},{\"question\":\"What multimodal data sources does the proposed framework use?\",\"answer\":\"It combines visual information (facial expression images and engagement-indicator images) with textual feedback from massive open online courses for sentiment analysis.\"},{\"question\":\"Which models are used for emotion, engagement, and sentiment detection?\",\"answer\":\"Convolutional neural networks (CNNs) are used for emotion and engagement classification, and a fine-tuned BERT model is used for sentiment analysis.\"}]","Multimodal Engagement and Sentiment Analytics in Health Science Education - A Learning Analytics Framework Integrating AI and Pedagogical Theory | PDF",1792034761,63]