[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-126587-en":3,"doc-seo-126587-105":31,"detail-sidebar-cat-0-en-105":92},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":21,"is_downloadable":21,"audit_status":21,"page_count":22,"language":23,"language_code":24,"site_id":25,"html_lang":24,"table_of_contents":26,"faqs":27,"seo_title":28,"seo_description":14,"update_tm":29,"read_time":30},126587,687207017582,"Himbo","https://ap-avatar.wpscdn.com/davatar_994ba38a5ba835b3df7d355c54d3ed8d",8,"Research & Report","In Situ Framework for Coupling Simulation and Machine Learning with Application to CFD","Machine learning is increasingly used to accelerate computational fluid dynamics, yet offline training creates major I/O and storage bottlenecks and requires complex runtime coupling between ML libraries and simulation codes. This work introduces an in situ approach that simplifies coupling and enables simultaneous training and inference on heterogeneous clusters. Using SmartSim, a database stores data and ML models in memory to bypass the file system. Experiments on Polaris show perfect scaling for transfer and inference costs, and an in situ autoencoder trained on turbulent flow adds negligible overhead compared to solver time steps and training epochs.","In Situ Framework for Coupling Simulation and Machine Learning with Application to CFD  \nRiccardo Balin  \n[rbalin@anl.gov](rbalin@anl.gov)[ ](rbalin@anl.gov)Argonne National Laboratory Lemont, IL, USA  \nFilippo Simini  \n[fsimini@anl.gov](fsimini@anl.gov)[ ](fsimini@anl.gov)Argonne National Laboratory Lemont, IL, USA  \nCooper Simpson  \n[cooper.simpson@colorado.edu](cooper.simpson@colorado.edu)[ ](cooper.simpson@colorado.edu)University of Colorado Boulder Boulder, CO, USA  \narXiv :2306 . 12900v1 [ cs .LG] 22 Jun 2023  \nAndrew Shao  \n[andrew.shao@hpe.com](andrew.shao@hpe.com)[ ](andrew.shao@hpe.com)Hewlett Packard Enterprise Seattle, WA, USA  \nAlessandro Rigazzi  \n[alessandro.rigazzi@hpe.com](alessandro.rigazzi@hpe.com)[ ](alessandro.rigazzi@hpe.com)Hewlett Packard Enterprise Switzerland  \nMatthew Ellis  \n[matthew.ellis@hpe.com](matthew.ellis@hpe.com)[ ](matthew.ellis@hpe.com)Hewlett Packard Enterprise Seattle, WA, USA  \nStephen Becker  \n[stephen.becker@colorado.edu](stephen.becker@colorado.edu)[ ](stephen.becker@colorado.edu)University of Colorado Boulder Boulder, CO, USA  \nAlireza Doostan  \n[alireza.doostan@colorado.edu](alireza.doostan@colorado.edu)[ ](alireza.doostan@colorado.edu)University of Colorado Boulder Boulder, CO, USA  \nJohn A. Evans  \n[john.a.evans@colorado.edu](john.a.evans@colorado.edu)[ ](john.a.evans@colorado.edu)University of Colorado Boulder Boulder, CO, USA  \nKenneth E. Jansen  \n[kenneth.jansen@colorado.edu](kenneth.jansen@colorado.edu)[ ](kenneth.jansen@colorado.edu)University of Colorado Boulder Boulder, CO, USA  \nABSTRACT  \nRecent years have seen many successful applications of machine learning (ML) to facilitate fluid dynamic computations. As simulations grow, generating new training datasets for traditional offline learning creates I/O and storage bottlenecks. Additionally, performing inference at runtime requires non-trivial coupling of ML framework libraries with simulation codes. This work offers a solution to both limitations by simplifying this coupling and enabling in situ training and inference workflows on heterogeneous clusters. Leveraging SmartSim, the presented framework deploys a database to store data and ML models in memory, thus circumventing the file system. On the Polaris supercomputer, we demonstrate perfect scaling efficiency to the full machine size of the data transfer and inference costs thanks to a novel co-located deployment of the database. Moreover, we train an autoencoder in situ from a turbulent flow simulation, showing that the framework overhead is negligible relative to a solver time step and training epoch.  \n1 INTRODUCTION  \nThe last decade has seen a number of successful demonstrations of the use of machine learning (ML) to facilitate computational fluid dynamic (CFD) simulations [4] . These works span a wide range of applications, including the development of turbulent closure models of various forms and fidelities [1, 22, 24, 33], compression of flow states [8, 12], turbulent inflow generation [39], flow control [16], and even surrogate modeling of the Navier-Stokes equations [23, 34] . In all these cases, the ML models were trained using the traditional offline (or post hoc) approach, wherein the training data is produced beforehand by a high-fidelity simulation, usually by a different research group [1, 24], and stored in private or public databases [6, 29] .  \n(a) In situ training.  \n(b) In situ inference.  \nFigure 1: Framework for in situ training and inference coupling CFD simulation codes to ML workloads with a SmartSim database and SmartRedis client library.  \nWhile more straightforward, the offline data pipeline imposes some important restrictions to the development and deployment of ML models which can generalize to new geometries and physical regimes. First, generating, storing and maintaining the training database is a costly exercise since it involves storing many high-fidelity solution snapshots computed with direct numerical simulations (DNS) [6, 29] . Se","cbCaidpy72HdDix9","https://ap.wps.com/l/cbCaidpy72HdDix9","pdf",1238071,3,1,11,"English","en",105,"# Introduction\n## Limits of offline ML pipelines for CFD\n## Benefits of in situ (online) training and inference\n## Coupling strategies: tightly-coupled vs loosely-coupled","[{\"question\":\"What problem does the proposed framework address in CFD machine learning workflows?\",\"answer\":\"It addresses two bottlenecks of offline ML for CFD: costly dataset generation/storage and the non-trivial coupling needed to run ML inference at runtime within simulation codes.\"},{\"question\":\"How does the framework enable in situ training and inference?\",\"answer\":\"It uses SmartSim to store data and ML models in memory, circumventing the file system and allowing training and inference workflows to run together with the simulation on heterogeneous clusters.\"},{\"question\":\"What results were demonstrated on the Polaris supercomputer?\",\"answer\":\"The framework achieved perfect scaling efficiency for the full machine size with respect to data transfer and inference costs, enabled by a novel co-located database deployment.\"}]","In Situ Framework for Coupling Simulation and Machine Learning with Application to CFD | PDF",1785933522,28,{"code":4,"msg":32,"data":33},"ok",{"site_id":25,"language":24,"slug":34,"title":13,"keywords":35,"description":14,"schema_data":36,"social_meta":87,"head_meta":89,"extra_data":91,"updated_unix":29},"in-situ-framework-for-coupling-simulation-and-machine-learning-with-application-to-cfd","",{"@graph":37,"@context":86},[38,54,69],{"@type":39,"itemListElement":40},"BreadcrumbList",[41,45,49,51],{"item":42,"name":43,"@type":44,"position":21},"https://docshare.wps.com","Home","ListItem",{"item":46,"name":47,"@type":44,"position":48},"https://docshare.wps.com/document/","Document",2,{"item":50,"name":12,"@type":44,"position":20},"https://docshare.wps.com/document/research-report/",{"item":52,"name":13,"@type":44,"position":53},"https://docshare.wps.com/document/in-situ-framework-for-coupling-simulation-and-machine-learning-with-application-to-cfd/126587/",4,{"url":52,"name":13,"@type":55,"author":56,"headline":13,"publisher":58,"fileFormat":61,"inLanguage":24,"description":14,"dateModified":62,"datePublished":63,"encodingFormat":61,"isAccessibleForFree":64,"interactionStatistic":65},"DigitalDocument",{"name":9,"@type":57},"Person",{"url":42,"name":59,"@type":60},"DocShare","Organization","application/pdf","2026-08-23","2026-08-05",true,{"@type":66,"interactionType":67,"userInteractionCount":20},"InteractionCounter",{"@type":68},"ViewAction",{"@type":70,"mainEntity":71},"FAQPage",[72,78,82],{"name":73,"@type":74,"acceptedAnswer":75},"What problem does the proposed framework address in CFD machine learning workflows?","Question",{"text":76,"@type":77},"It addresses two bottlenecks of offline ML for CFD: costly dataset generation/storage and the non-trivial coupling needed to run ML inference at runtime within simulation codes.","Answer",{"name":79,"@type":74,"acceptedAnswer":80},"How does the framework enable in situ training and inference?",{"text":81,"@type":77},"It uses SmartSim to store data and ML models in memory, circumventing the file system and allowing training and inference workflows to run together with the simulation on heterogeneous clusters.",{"name":83,"@type":74,"acceptedAnswer":84},"What results were demonstrated on the Polaris supercomputer?",{"text":85,"@type":77},"The framework achieved perfect scaling efficiency for the full machine size with respect to data transfer and inference costs, enabled by a novel co-located database deployment.","https://schema.org",{"og:url":52,"og:type":88,"og:title":13,"og:site_name":59,"og:description":14},"article",{"robots":90,"canonical":52},"index,follow",{"doc_id":7,"site_id":25},{"code":4,"msg":5,"data":93},[94,98,102,106,111,116,121,124,129,132,136],{"id":21,"doc_module":4,"doc_module_name":47,"category_name":95,"show_sort_weight":96,"slug":97},"Story & Novel",90,"story-novel",{"id":48,"doc_module":4,"doc_module_name":47,"category_name":99,"show_sort_weight":100,"slug":101},"Literature",80,"literature",{"id":53,"doc_module":4,"doc_module_name":47,"category_name":103,"show_sort_weight":104,"slug":105},"Exam",70,"exam",{"id":107,"doc_module":4,"doc_module_name":47,"category_name":108,"show_sort_weight":109,"slug":110},5,"Comic",60,"comic",{"id":112,"doc_module":4,"doc_module_name":47,"category_name":113,"show_sort_weight":114,"slug":115},6,"Technology",50,"technology",{"id":117,"doc_module":4,"doc_module_name":47,"category_name":118,"show_sort_weight":119,"slug":120},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":47,"category_name":12,"show_sort_weight":122,"slug":123},30,"research-report",{"id":125,"doc_module":4,"doc_module_name":47,"category_name":126,"show_sort_weight":127,"slug":128},9,"Religion & Spirituality",20,"religion-spirituality",{"id":127,"doc_module":4,"doc_module_name":47,"category_name":130,"show_sort_weight":127,"slug":131},"World Cup","world-cup",{"id":133,"doc_module":4,"doc_module_name":47,"category_name":134,"show_sort_weight":133,"slug":135},10,"Lifestyle","lifestyle",{"id":137,"doc_module":4,"doc_module_name":47,"category_name":138,"show_sort_weight":107,"slug":139},19,"General","general"]