[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-124607-en":3,"doc-seo-124607-105":30,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":4,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":27,"seo_description":14,"update_tm":28,"read_time":29},124607,687197100911,"Himbo","https://ap-avatar.wpscdn.com/avatar/a000239b6f1da00475?x-image-process=image/resize,m_fixed,w_180,h_180&k=1785132997149421697",8,"Research & Report","Leveraging machine learning to examine engagement with a digital therapeutic - Brief Research Report","Digital Therapeutics (DTx) are evidence-based software interventions that can capture high-temporal-precision data on both the quantity and the quality of patient engagement. Because treatment success may depend on how users engage—especially for cognitive interventions—this report introduces a near-real-time technique to evaluate interaction quality. The approach scores engagement at an approximately four-minute mission level by classifying whether users use the treatment as intended, using SME-labeled data and achieving strong test performance (accuracy and F1 of 0.94).","TYPE Brief Research Report PUBLISHED 02 June 2023  \nDOI 10.3389/fdgth.2023.1063165  \nEDITED BY  \nKirsten Smayda,  \nMedRhythms, United States  \nREVIEWED BY  \nAngel Enrique Roig, Silvercloud Health, Ireland Anis Davoudi,  \nUniversity of Florida, United States  \n*CORRESPONDENCE  \nAndrew C. Heusser  \n [aheusser@akiliinteractive.com](aheusser@akiliinteractive.com)  \n†These authors have contributed equally to this work and share first authorship  \nRECEIVED 06 October 2022  \nACCEPTED 10 May 2023  \nPUBLISHED 02 June 2023  \nCITATION  \nHeusser AC, DeLoss DJ, Cañadas E and Alailima T (2023) Leveraging machine learning to examine engagement with a digital therapeutic.  \nFront. Digit. Health 5:1063165 .  \ndoi: 10.3389/fdgth.2023.1063165  \nCOPYRIGHT  \n© 2023 Heusser, DeLoss, Cañadas and Alailima. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY) . The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.  \nLeveraging machine learning to examine engagement with a digital therapeutic  \nAndrew C. Heusser*†, Denton J. DeLoss†, Elena Cañadasand Titiimaea Alailima  \nAkili Interactive, Boston, MA, United States  \nDigital Therapeutics (DTx) are evidence-based software-driven interventions for the prevention, management, and treatment of medical disorders or diseases. DTx offer the unique ability to capture rich objective data about when and how a patient engages with a treatment. Not only can one measure the quantity of patient interactions with a digital treatment with high temporal precision, but one can also assess the quality of these interactions. This is particularly useful for treatments such as cognitive interventions, where the specific manner in which a patient engages may impact likelihood of treatment success. Here, we present a technique for measuring the quality of user interactions with a digital treatment in near-real time. This approach produces evaluations at the level of a roughly four-minute gameplay session (mission) . Each mission required users to engage in adaptive and personalized multitasking training. The training included simultaneous presentation of a sensory-motor navigation task and a perceptual discrimination task. We trained a machine learning model to classify user interactions with the digital treatment to determine if they were “using it as intended” or “not using it as intended” based on labeled data created by subject matter experts (SME) . On a held-out test set, the classifier was able to reliably predict the SME-derived labels (Accuracy = . 94; F1 Score = . 94) . We discuss the value of this approach and highlight exciting future directions for shared decision-making and communication between caregivers, patients and healthcare providers. Additionally, the output of this technique can be useful for clinical trials and personalized intervention.  \nKEYWORDS  \ndigital therapeutics, engagement, machine learning, cognition, brain health  \nIntroduction  \nDigital mental health interventions target the prevention or treatment of mental health disorders and associated impairments (i.e., functional, affective, cognitive) delivered via a digital platform (e.g., web browser, mobile apps, text messaging, or virtual reality) (1) . They offer the potential to overcome availability and accessibility limitations, including geographical location and time (2–4) .  \nWhile there are thousands of digital interventions claiming to improve various aspects of mental health, many of them have never gone through clinical trials or regulatory scrutiny. Also, due to a number of factors, including fast growth of the industry and an absence of well-accepted standards, there are widely varying definitions of w","cbCaip3H8c5iLK4g","https://ap.wps.com/l/cbCaip3H8c5iLK4g","pdf",715219,1,6,"English","en",105,"# Introduction\n## Digital mental health and DTx\n## Engagement and why quality matters\n## Limits of adherence and retention measures","[{\"question\":\"What are digital therapeutics (DTx) and why do they matter for measuring engagement?\",\"answer\":\"DTx are evidence-based software-driven interventions that record objective data on when and how patients engage. This enables assessment of both interaction quantity and interaction quality with medical treatments.\"},{\"question\":\"How does the proposed technique evaluate engagement quality?\",\"answer\":\"It trains a machine learning model to classify user interactions as “using it as intended” or “not using it as intended” based on SME-labeled data. The evaluation is produced near-real time at roughly a four-minute mission session level.\"},{\"question\":\"How well does the classifier perform on the test set?\",\"answer\":\"On a held-out test set, the classifier reliably predicts the SME-derived labels with accuracy and F1 score both reported as 0.94.\"}]","Leveraging machine learning to examine engagement with a digital therapeutic - Brief Research Report | PDF",1785893287,15,{"code":4,"msg":31,"data":32},"ok",{"site_id":24,"language":23,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":28},"leveraging-machine-learning-to-examine-engagement-with-a-digital-therapeutic-brief-research-report","",{"@graph":36,"@context":85},[37,54,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,51],{"item":41,"name":42,"@type":43,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":50},"https://docshare.wps.com/document/research-report/",3,{"item":52,"name":13,"@type":43,"position":53},"https://docshare.wps.com/document/leveraging-machine-learning-to-examine-engagement-with-a-digital-therapeutic-brief-research-report/124607/",4,{"url":52,"name":13,"@type":55,"author":56,"headline":13,"publisher":58,"fileFormat":61,"inLanguage":23,"description":14,"dateModified":62,"datePublished":62,"encodingFormat":61,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":57},"Person",{"url":41,"name":59,"@type":60},"DocShare","Organization","application/pdf","2026-08-05",true,{"@type":65,"interactionType":66,"userInteractionCount":4},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What are digital therapeutics (DTx) and why do they matter for measuring engagement?","Question",{"text":75,"@type":76},"DTx are evidence-based software-driven interventions that record objective data on when and how patients engage. This enables assessment of both interaction quantity and interaction quality with medical treatments.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"How does the proposed technique evaluate engagement quality?",{"text":80,"@type":76},"It trains a machine learning model to classify user interactions as “using it as intended” or “not using it as intended” based on SME-labeled data. The evaluation is produced near-real time at roughly a four-minute mission session level.",{"name":82,"@type":73,"acceptedAnswer":83},"How well does the classifier perform on the test set?",{"text":84,"@type":76},"On a held-out test set, the classifier reliably predicts the SME-derived labels with accuracy and F1 score both reported as 0.94.","https://schema.org",{"og:url":52,"og:type":87,"og:title":13,"og:site_name":59,"og:description":14},"article",{"robots":89,"canonical":52},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,114,119,122,127,130,134],{"id":20,"doc_module":4,"doc_module_name":46,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":53,"doc_module":4,"doc_module_name":46,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":46,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":21,"doc_module":4,"doc_module_name":46,"category_name":111,"show_sort_weight":112,"slug":113},"Technology",50,"technology",{"id":115,"doc_module":4,"doc_module_name":46,"category_name":116,"show_sort_weight":117,"slug":118},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":120,"slug":121},30,"research-report",{"id":123,"doc_module":4,"doc_module_name":46,"category_name":124,"show_sort_weight":125,"slug":126},9,"Religion & Spirituality",20,"religion-spirituality",{"id":125,"doc_module":4,"doc_module_name":46,"category_name":128,"show_sort_weight":125,"slug":129},"World Cup","world-cup",{"id":131,"doc_module":4,"doc_module_name":46,"category_name":132,"show_sort_weight":131,"slug":133},10,"Lifestyle","lifestyle",{"id":135,"doc_module":4,"doc_module_name":46,"category_name":136,"show_sort_weight":106,"slug":137},19,"General","general"]