[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-85414-en":3,"doc-seo-85414-105":29,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},85414,1099513958762,"Logic","https://ap-avatar.wpscdn.com/avatar/1000023916a998db790?x-image-process=image/resize,m_fixed,w_180,h_180&k=1782109480056885918",8,"Research & Report","BiomechGPT: Extending Motion-Language Models to Clinical Motion Understanding","Markerless motion capture enables high-quality biomechanical data in clinical settings, but scalable analysis for diverse downstream questions remains difficult. BiomechGPT studies whether multimodal motion–language models can answer detailed, clinically meaningful questions using language-based interaction. The work trains on 71 hours of biomechanical data from 750 participants, then expands training via a cross-format tokenizer that encodes heterogeneous motion formats into a shared latent space without paired samples, enabling pooled question–answer supervision. BiomechGPT delivers competitive performance across clinically relevant tasks with gains from dataset and model scale.","arXiv :2505 . 18465v2 [ cs .CV] 10 Jul 2026  \nBiomechGPT: Extending Motion-Language Models to Clinical Motion Understanding  \nRuize Yang1 ,2 , Ann Kennedy3 , R. James Cotton1 ,4  \n1 Center for Bionic Medicine, Shirley Ryan AbilityLab, Chicago, IL 2 Department of Neuroscience, Northwestern University, Chicago, IL 3 Department of Neuroscience, The Scripps Research Institute, San Deigo, CA 4 Department of Physical Medicine & Rehabilitation, Northwestern University, Chicago, IL  \nAdvances in markerless motion capture are making high-quality biomechanical data increasingly accessible, creating a growing need for scalable downstream analytics. Building a bespoke pipeline for each analysis task is time-consuming, motivating models that can flexibly handle diverse clinical questions within a single framework. Recent work has shown that fine-tuning language models to accept tokenized motion as an additional modality enables descriptive captioning of movement, raising the question of whether these models are also capable of clinically relevant motion understanding, where diverse tasks and annotations provide a natural testbed. We investigate whether such a multimodal motion–language model can answer detailed, clinically meaningful questions about movement. We collected 71 hours of biomechanical data from 750 participants, many with movement impairments, performing tasks commonly used in clinical assessment. To further expand the training dataset, we designed a cross-format tokenizer that directly encodes motion data from heterogeneous formats into a shared latent space without paired data, allowing a second dataset to be incorporated and enabling pooling annotations across datasets. From these tokenized representations, we constructed a multimodal dataset of motion-related question–answer pairs and used it to train BiomechGPT, a multimodal biomechanics–language model. BiomechGPT achieves competitive performance across a range of clinically relevant tasks, with performance scaling with both dataset and model size. It offers a new way for clinicians and researchers to interact with biomechanical data and represents a promising direction for rehabilitation-focused movement analysis. Project page: [https://intelligentsensingandrehabilitation.github.io/BiomechGPT/](https://intelligentsensingandrehabilitation.github.io/BiomechGPT/)  \nCorrespondence: [rcotton@sralab.org](rcotton@sralab.org)  \nISR  \nIntelligent Sensing and Rehabilitation  \nKeywords: Biomechanics, gait analysis, large language models, machine learning, rehabilitation  \n1 Introduction  \nHow someone moves carries rich information about their health (Baker, 2006) . Recent advances in markerless motion capture have made it substantially easier to capture biomechanics in clinically accessible settings (Kanko et al. , 2021; Uhlrich et al. , 2023) . Movement can now be recorded in several ways, ranging from multi-camera systems (Cotton, 2024) to a single smartphone camera (Peiffer et al. , 2026) . While this accessibility lets clinicians and scientists measure more aspects of movement, a remaining barrier to clinical impact is analyzing the resulting data. People perform a  \n Model flow  Forward kinemetics  Site similarity loss  Concatenate  \n\n|  | \u003Cbr> |\n| --- | --- |\n\n\n|  | \u003Cbr> |  |  |\n| --- | --- | --- | --- |\n|  |  |  |  |\n\nFigure 1: Overview of BiomechGPT. Stage 1: Cross-format tokenizer training. We trained a VQ-VAE-based tokenizer jointly on the Clinical dataset (in biomechanical model (Biom) format) and the HumanML3D dataset (in SMPL format) . Format-specific encoders embed motion from either format into a shared motion codebook, and format-specific decoders reconstruct motion in both formats from the quantized tokens. A site-similarity loss compares site locations between the input motion and the cross-format decoded output (obtained via forward kinematics), enforcing the shared latent space without requiring paired samples across formats. Stage 2: BiomechGPT training. We fro","cbCaifHtEOHK7uIh","https://ap.wps.com/l/cbCaifHtEOHK7uIh","pdf",4152890,1,28,"English","en",105,"# Introduction\n## Clinical relevance of movement data\n## Motivation for unified motion analysis\n## Tokenizer-based motion–language modeling","[{\"question\":\"What problem does BiomechGPT address in clinical movement analysis?\",\"answer\":\"It targets the difficulty of scaling analytics when many clinical questions and annotations must be handled, avoiding bespoke pipelines for each task.\"},{\"question\":\"How is the training data constructed and expanded for BiomechGPT?\",\"answer\":\"The model uses 71 hours of biomechanical data from 750 participants for clinical assessment tasks and expands coverage by pooling datasets through a cross-format tokenizer.\"},{\"question\":\"What does the cross-format tokenizer enable?\",\"answer\":\"It directly encodes heterogeneous motion formats into a shared latent space without requiring paired samples, allowing annotations from different sources to be pooled for training.\"}]",1784203224,71,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":27},"biomechgpt-extending-motion-language-models-to-clinical-motion-understanding","",{"@graph":35,"@context":85},[36,53,68],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/biomechgpt-extending-motion-language-models-to-clinical-motion-understanding/85414/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-17","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What problem does BiomechGPT address in clinical movement analysis?","Question",{"text":75,"@type":76},"It targets the difficulty of scaling analytics when many clinical questions and annotations must be handled, avoiding bespoke pipelines for each task.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"How is the training data constructed and expanded for BiomechGPT?",{"text":80,"@type":76},"The model uses 71 hours of biomechanical data from 750 participants for clinical assessment tasks and expands coverage by pooling datasets through a cross-format tokenizer.",{"name":82,"@type":73,"acceptedAnswer":83},"What does the cross-format tokenizer enable?",{"text":84,"@type":76},"It directly encodes heterogeneous motion formats into a shared latent space without requiring paired samples, allowing annotations from different sources to be pooled for training.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":45,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":45,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":45,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":45,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":45,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":45,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":45,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]