[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-118599-en":3,"doc-seo-118599-105":29,"detail-sidebar-cat-0-en-105":90},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":4,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":11,"language":21,"language_code":22,"site_id":23,"html_lang":22,"table_of_contents":24,"faqs":25,"seo_title":26,"seo_description":14,"update_tm":27,"read_time":28},118599,1374391974585,"Genevieve","https://ap-avatar.wpscdn.com/davatar_276721f389ce27ea32af1340a28f341c",8,"Research & Report","Towards Relevant Human-Vehicle Interaction Data for Perceptive Machine Learning","Machine learning models for open-world tasks like autonomous driving depend on large, diverse datasets, yet common situations dominate while rare scenarios and edge cases remain underrepresented, limiting performance. The work introduces a simulation framework that uses human-in-the-loop closed-loop virtual reality combined with full-body motion capture to record scene-relevant interactions between humans and virtual objects and to produce realistic ground truths. An action recognition model trained on the simulated data must decide whether a vehicle is the intended target of a waving pedestrian, showing the value of spatial and contextual encodings and demonstrating improved realism and diversity for complex human behavior.","Towards relevant human-vehicle interaction data for perceptive machine learning  \nMarkus Rehmann, Michael Brunner, and Cristbal Curio  \nFaculty of Informatics  \nReutlingen University  \n72762 Reutlingen, Germany  \n{markus.rehmann, michael.brunner, [cristobal.curio](cristobal.curio}@reutlingen-university.de)[}](cristobal.curio}@reutlingen-university.de)[@reutlingen-university.de](cristobal.curio}@reutlingen-university.de)  \nAbstract—Machine learning models, particularly those used in open-world settings like autonomous driving, require extensive and diverse datasets for effective training. However, the overabundance of standard situations in large datasets often leads to underrepresentation of rare scenarios and edge cases, which are crucial for achieving high performance across all situations. This work proposes a simulation framework for capturing human-in-the-loop simulation data, enabling the creation of realistic and diverse datasets for machine learning models as well as human behavior studies. The framework combines full body motion capture with virtual reality to record scene-relevant interactions between humans and virtual objects, allowing for the generation of various ground truthsin simulation. To validate the effectiveness of this approach, an action recognition model is trained on simulated data generated by the proposed framework. From the point of view of a vehicle, the model needs to determine if the vehicle is the intended target of a waving pedestrian. The investigation into different keypoint representations and spatial encodings reveals the importance of integrating spatial and contextual data for accurate action recognition. Furthermore, the results show that human-in-the-loop simulations can effectively capture complex human behaviors and interactions, enabling the creation of more realistic and diverse datasets for machine learning models.  \nIndex Terms—simulation, human-in-the-loop, interaction, virtual reality, machine learning, action recognition.  \nI. INTRODUCTION  \nMachine learning requires much data for training. The tendency is to gather as much data as possible, which often results in very big datasets. Due to the overabundance of standard situations, rarely occurring scenarios and edge cases may only be barely represented in the recorded dataset, reducing the performance of any machine learning model for such situations. To achieve similar performance on all situations in a dataset, the available data should be uniformly distributed. This leads to the idea that better distributed and thus more relevant data should be used to train neural networks. However, the rare occurrence of edge cases in  \nThe research leading to these results is funded by the German Federal Ministry for Economic Affairs and Climate Action within the project ’KI Delta Learning’(grant 19A19013S) and by the European Union’s Horizon Europe research and innovation programme within the project ’HEIDI’(grant 101069538) . Views and opinions expressed are those of the author(s) only and do not necessarily reflect those of the European Union. Neither the European Union nor the granting authority can be held responsible for them.  \ndatasets often has a reason. Most edge cases are difficult to capture, recreate or simply dangerous in the real world. In simulations, many of these downsides do not exist, and they offer further advantages. For instance, annotations do not contain inaccuracies and biases of human annotators, even for data that cannot feasibly be annotated by humans, such as radar and depth data. Simulations allow to recreate and evaluate real-world or completely new edge cases. Dangerous situations in the real world can be safely tested and simulated scenarios are reproducible. There is also no need for privacy workarounds in camera data like blurring.  \nMany machine learning models are designed for openworld settings, such as autonomous driving, where anything can happen. To achieve acceptable safety and reliability level","cbCaigltnLqc1bzo","https://ap.wps.com/l/cbCaigltnLqc1bzo","pdf",4028426,1,"English","en",105,"# Introduction\n# Related Work","[{\"question\":\"Why are rare scenarios and edge cases important for perceptive machine learning in autonomous driving?\",\"answer\":\"They are crucial for achieving consistently high performance across all situations. Standard data collections often overrepresent common cases and underrepresent rare events, reducing model reliability in those conditions.\"},{\"question\":\"What does the proposed simulation framework combine to capture human-vehicle interaction data?\",\"answer\":\"It combines full-body optical motion capture with a virtual reality, closed-loop human-in-the-loop workflow. This records realistic interactions between humans and virtual objects and supports ground truth generation.\"},{\"question\":\"How is the effectiveness of the framework validated?\",\"answer\":\"An action recognition model is trained using simulated data from the framework. The vehicle-side task is to determine whether it is the intended target of a waving pedestrian, and results highlight the importance of spatial and contextual representations.\"}]","Towards Relevant Human-Vehicle Interaction Data for Perceptive Machine Learning | PDF",1785684446,20,{"code":4,"msg":30,"data":31},"ok",{"site_id":23,"language":22,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":85,"head_meta":87,"extra_data":89,"updated_unix":27},"towards-relevant-human-vehicle-interaction-data-for-perceptive-machine-learning","",{"@graph":35,"@context":84},[36,53,67],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/towards-relevant-human-vehicle-interaction-data-for-perceptive-machine-learning/118599/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":22,"description":14,"dateModified":61,"datePublished":61,"encodingFormat":60,"isAccessibleForFree":62,"interactionStatistic":63},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-08-02",true,{"@type":64,"interactionType":65,"userInteractionCount":4},"InteractionCounter",{"@type":66},"ViewAction",{"@type":68,"mainEntity":69},"FAQPage",[70,76,80],{"name":71,"@type":72,"acceptedAnswer":73},"Why are rare scenarios and edge cases important for perceptive machine learning in autonomous driving?","Question",{"text":74,"@type":75},"They are crucial for achieving consistently high performance across all situations. Standard data collections often overrepresent common cases and underrepresent rare events, reducing model reliability in those conditions.","Answer",{"name":77,"@type":72,"acceptedAnswer":78},"What does the proposed simulation framework combine to capture human-vehicle interaction data?",{"text":79,"@type":75},"It combines full-body optical motion capture with a virtual reality, closed-loop human-in-the-loop workflow. This records realistic interactions between humans and virtual objects and supports ground truth generation.",{"name":81,"@type":72,"acceptedAnswer":82},"How is the effectiveness of the framework validated?",{"text":83,"@type":75},"An action recognition model is trained using simulated data from the framework. The vehicle-side task is to determine whether it is the intended target of a waving pedestrian, and results highlight the importance of spatial and contextual representations.","https://schema.org",{"og:url":51,"og:type":86,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":88,"canonical":51},"index,follow",{"doc_id":7,"site_id":23},{"code":4,"msg":5,"data":91},[92,96,100,104,109,114,119,122,126,129,133],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":93,"show_sort_weight":94,"slug":95},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":97,"show_sort_weight":98,"slug":99},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":101,"show_sort_weight":102,"slug":103},"Exam",70,"exam",{"id":105,"doc_module":4,"doc_module_name":45,"category_name":106,"show_sort_weight":107,"slug":108},5,"Comic",60,"comic",{"id":110,"doc_module":4,"doc_module_name":45,"category_name":111,"show_sort_weight":112,"slug":113},6,"Technology",50,"technology",{"id":115,"doc_module":4,"doc_module_name":45,"category_name":116,"show_sort_weight":117,"slug":118},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":120,"slug":121},30,"research-report",{"id":123,"doc_module":4,"doc_module_name":45,"category_name":124,"show_sort_weight":28,"slug":125},9,"Religion & Spirituality","religion-spirituality",{"id":28,"doc_module":4,"doc_module_name":45,"category_name":127,"show_sort_weight":28,"slug":128},"World Cup","world-cup",{"id":130,"doc_module":4,"doc_module_name":45,"category_name":131,"show_sort_weight":130,"slug":132},10,"Lifestyle","lifestyle",{"id":134,"doc_module":4,"doc_module_name":45,"category_name":135,"show_sort_weight":105,"slug":136},19,"General","general"]