[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-86347-en":3,"doc-seo-86347-105":29,"detail-sidebar-cat-0-en-105":90},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":21,"is_downloadable":21,"audit_status":21,"page_count":11,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},86347,1099513958607,"Jiven","https://ap-avatar.wpscdn.com/avatar/100002390cf8733938c?x-image-process=image/resize,m_fixed,w_180,h_180&k=1778829742770036399",8,"Research & Report","Constrained Reinforcement Learning for Safe Heat Pump Control","Constrained reinforcement learning for building heat pump control: minimize electrical energy while keeping indoor temperature within comfort bounds. The task is formulated as a constrained Markov decision process with explicit comfort constraints, motivated by the observation that optimal operation typically concentrates near the feasibility boundary. A smoothed log-barrier variant of Soft Actor-Critic (CSAC-LB) leverages this structure and is evaluated for robustness under sensor noise and model mismatch. The paper introduces I4B, a lightweight Gym-style simulator with an MPC baseline, and shows CSAC-LB matches or surpasses MPC while maintaining constraint satisfaction across two scenarios.","Constrained Reinforcement Learning for Safe Heat Pump Control  \nBaohe Zhang∗†, Lilli Frison∗§, Thomas Brox†, Joschka Bödecker†  \narXiv :2409 . 197 16v2 [ cs .LG] 13 Jul 2026  \nAbstract—We study heat pump control in buildings as a constrained reinforcement learning (RL) problem: minimize electrical energy while keeping indoor temperature within comfort bounds. We formulate the task as a constrained Markov decision process with temperature comfort constraints, argue adn verify that optimal operation typically lies near the comfort boundary and that a smoothed log-barrier variant of Soft Actor-Critic (CSAC-LB) exploits this structure, and report robustness to sensor noise and model mismatch. We release I4B, a lightweight simulator with a Gym-style API and a built-in MPC baseline to enable reproducible RL studies. On two building scenarios, CSAC-LB balances comfort and energy at least on par with MPC while maintaining constraint satisfaction. Benchmarking against several baseline algorithms demonstrates CSAC-LB’s efficiency in exploration and control performance.  \nI. INTRODUCTION  \nHeat pumps (HP) have become the central component of modern building heating systems, offering high efficiency but requiring precise control to balance comfort and energy use. Constrained RL methods can balance thermal comfort with energy usage while learning from noisy sensor data in realtime, as demonstrated in robotic tasks [1] . They also enable continuous learning in dynamic scenarios where building usage or electricity prices change.  \nSimulation frameworks play a vital role in training RL agents. An accurate, adaptable, and parallelizable simulator can significantly speed up training, leading to more precise control over heating systems. Existing frameworks for heating control lack a comprehensive, open-source, and lightweight simulator that combines multiple building environments, flexible customization and explicit HP actuator control. To fill this gap, we propose Intelligence for Building (I4B), a novel open-source framework for advanced HP control strategies like model predictive control (MPC) and RL for heat pump operation. I4B provides an interface between the building simulation module and control algorithms, incorporating reference controllers, support for parallelization, and standardized metrics for evaluation.  \nWe frame the heating control problem as a constrained Markov Decision Process (CMDP), aiming to minimize energy usage while maintaining indoor temperature above a set threshold. By applying state-of-the-art constrained RL algorithms within this framework, we benchmark their performance across various scenarios. Notably, the constrained RL with linear smoothed log barrier function (CSAC-LB) proves particularly suitable for heating control problems  \n∗ : These authors contributed equally  \n†: Faculty of Computer Science, University of Freiburg, Germany. § : Systems Control and Optimization Laboratory, Department of Microsystems Engineering, University of Freiburg, Germany. Contact Address: [zhangb@cs.uni-freiburg.de](zhangb@cs.uni-freiburg.de)  \ncharacterized by optimal solutions at the boundary of feasible and infeasible sets. Our findings show that CSAC-LB excelsin balancing exploration and performance. Our contributions are as follows:  \n• We propose I4B 1 , a new open-source lightweight building heat pump operation simulator with rich customization options and interfaces for different research communities.  \n• We apply a variety of constrained RL algorithms to different heating scenarios. An empirical study is performed to benchmark them. Code and experiments are publicly available.  \n• We demonstrate that CSAC-LB can balance the objective and constraints better compared to other SOTA methods, while achieving efficient exploration.  \nII. RELATED WORK  \nA. RL in Building Heating Control  \nRL has emerged as a promising approach for optimizing HVAC systems, achieving energy savings of 5–12% over standard controls [2], [3] . Res","cbCaiePimx5wzyjr","https://ap.wps.com/l/cbCaiePimx5wzyjr","pdf",933938,3,1,"English","en",105,"# I. INTRODUCTION\n# II. RELATED WORK\n## A. RL in Building Heating Control\n## B. Constrained RL Algorithms\n## C. Building Simulators","[{\"question\":\"How is the heat pump control problem modeled in the paper?\",\"answer\":\"It is formulated as a constrained Markov decision process that minimizes electrical energy while enforcing indoor temperature comfort bounds.\"},{\"question\":\"What is CSAC-LB and why is it used here?\",\"answer\":\"CSAC-LB is a smoothed log-barrier variant of Soft Actor-Critic designed to exploit the structure where optimal operation lies near the comfort feasibility boundary.\"},{\"question\":\"What is I4B and how does it support reproducible research?\",\"answer\":\"I4B is a lightweight, Gym-style simulator for heat pump operation, including reference controllers and an MPC baseline, enabling standardized metrics and reproducible constrained RL studies.\"}]",1784210713,20,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":85,"head_meta":87,"extra_data":89,"updated_unix":27},"constrained-reinforcement-learning-for-safe-heat-pump-control","",{"@graph":35,"@context":84},[36,52,67],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,49],{"item":40,"name":41,"@type":42,"position":21},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":20},"https://docshare.wps.com/document/research-report/",{"item":50,"name":13,"@type":42,"position":51},"https://docshare.wps.com/document/constrained-reinforcement-learning-for-safe-heat-pump-control/86347/",4,{"url":50,"name":13,"@type":53,"author":54,"headline":13,"publisher":56,"fileFormat":59,"inLanguage":23,"description":14,"dateModified":60,"datePublished":61,"encodingFormat":59,"isAccessibleForFree":62,"interactionStatistic":63},"DigitalDocument",{"name":9,"@type":55},"Person",{"url":40,"name":57,"@type":58},"DocShare","Organization","application/pdf","2026-07-23","2026-07-16",true,{"@type":64,"interactionType":65,"userInteractionCount":20},"InteractionCounter",{"@type":66},"ViewAction",{"@type":68,"mainEntity":69},"FAQPage",[70,76,80],{"name":71,"@type":72,"acceptedAnswer":73},"How is the heat pump control problem modeled in the paper?","Question",{"text":74,"@type":75},"It is formulated as a constrained Markov decision process that minimizes electrical energy while enforcing indoor temperature comfort bounds.","Answer",{"name":77,"@type":72,"acceptedAnswer":78},"What is CSAC-LB and why is it used here?",{"text":79,"@type":75},"CSAC-LB is a smoothed log-barrier variant of Soft Actor-Critic designed to exploit the structure where optimal operation lies near the comfort feasibility boundary.",{"name":81,"@type":72,"acceptedAnswer":82},"What is I4B and how does it support reproducible research?",{"text":83,"@type":75},"I4B is a lightweight, Gym-style simulator for heat pump operation, including reference controllers and an MPC baseline, enabling standardized metrics and reproducible constrained RL studies.","https://schema.org",{"og:url":50,"og:type":86,"og:title":13,"og:site_name":57,"og:description":14},"article",{"robots":88,"canonical":50},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":91},[92,96,100,104,109,114,119,122,126,129,133],{"id":21,"doc_module":4,"doc_module_name":45,"category_name":93,"show_sort_weight":94,"slug":95},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":97,"show_sort_weight":98,"slug":99},"Literature",80,"literature",{"id":51,"doc_module":4,"doc_module_name":45,"category_name":101,"show_sort_weight":102,"slug":103},"Exam",70,"exam",{"id":105,"doc_module":4,"doc_module_name":45,"category_name":106,"show_sort_weight":107,"slug":108},5,"Comic",60,"comic",{"id":110,"doc_module":4,"doc_module_name":45,"category_name":111,"show_sort_weight":112,"slug":113},6,"Technology",50,"technology",{"id":115,"doc_module":4,"doc_module_name":45,"category_name":116,"show_sort_weight":117,"slug":118},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":120,"slug":121},30,"research-report",{"id":123,"doc_module":4,"doc_module_name":45,"category_name":124,"show_sort_weight":28,"slug":125},9,"Religion & Spirituality","religion-spirituality",{"id":28,"doc_module":4,"doc_module_name":45,"category_name":127,"show_sort_weight":28,"slug":128},"World Cup","world-cup",{"id":130,"doc_module":4,"doc_module_name":45,"category_name":131,"show_sort_weight":130,"slug":132},10,"Lifestyle","lifestyle",{"id":134,"doc_module":4,"doc_module_name":45,"category_name":135,"show_sort_weight":105,"slug":136},19,"General","general"]