[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"detail-sidebar-cat-1-en-105":3,"doc-seo-240351-105":53,"doc-detail-240351-en":126},{"code":4,"msg":5,"data":6},0,"success",[7,14,19,24,29,34,39,44,49],{"id":8,"doc_module":9,"doc_module_name":10,"category_name":11,"show_sort_weight":12,"slug":13},11,1,"Template","Presentations",90,"presentations",{"id":15,"doc_module":9,"doc_module_name":10,"category_name":16,"show_sort_weight":17,"slug":18},12,"Resumes",80,"resumes",{"id":20,"doc_module":9,"doc_module_name":10,"category_name":21,"show_sort_weight":22,"slug":23},14,"Invoices",70,"invoices",{"id":25,"doc_module":9,"doc_module_name":10,"category_name":26,"show_sort_weight":27,"slug":28},15,"Posters",60,"posters",{"id":30,"doc_module":9,"doc_module_name":10,"category_name":31,"show_sort_weight":32,"slug":33},16,"Social Media",50,"social-media",{"id":35,"doc_module":9,"doc_module_name":10,"category_name":36,"show_sort_weight":37,"slug":38},17,"Forms",40,"forms",{"id":40,"doc_module":9,"doc_module_name":10,"category_name":41,"show_sort_weight":42,"slug":43},18,"Letters",30,"letters",{"id":45,"doc_module":9,"doc_module_name":10,"category_name":46,"show_sort_weight":47,"slug":48},21,"Paper Templates",5,"papers-templates",{"id":50,"doc_module":9,"doc_module_name":10,"category_name":51,"show_sort_weight":4,"slug":52},158,"General","general-158",{"code":4,"msg":54,"data":55},"ok",{"site_id":56,"language":57,"slug":58,"title":59,"keywords":60,"description":61,"schema_data":62,"social_meta":119,"head_meta":121,"extra_data":123,"updated_unix":125},105,"en","ems-pausing-during-database-deployment-lesson-learned","EMS Pausing During Database Deployment - Lesson Learned","","EMS outage incidents occurred during energy management system (EMS) database deployment across multiple entities, all involving the same EMS vendor. The failures left front-end processor (FEP) and SCADA databases paused, causing loss of control center visibility and inability to update EMS screens. Root causes included disconnected or restarting critical processes without adequate monitoring or alarms, and promotion steps halting after pausing critical data. Corrective actions focused on vendor-assisted recovery, syncing SCADA, restarting services, unpausing databases, and improving vendor product enhancements.",{"@graph":63,"@context":118},[64,80,101],{"@type":65,"itemListElement":66},"BreadcrumbList",[67,71,74,77],{"item":68,"name":69,"@type":70,"position":9},"https://docshare.wps.com","Home","ListItem",{"item":72,"name":10,"@type":70,"position":73},"https://docshare.wps.com/template/",2,{"item":75,"name":51,"@type":70,"position":76},"https://docshare.wps.com/template/general/",3,{"item":78,"name":59,"@type":70,"position":79},"https://docshare.wps.com/template/ems-pausing-during-database-deployment-lesson-learned/240351/",4,{"url":78,"name":59,"@type":81,"image":82,"author":87,"headline":59,"publisher":90,"fileFormat":93,"inLanguage":57,"description":61,"dateModified":94,"datePublished":95,"encodingFormat":93,"isAccessibleForFree":96,"interactionStatistic":97},"DigitalDocument",{"url":83,"@type":84,"width":85,"height":86},"https://docshare.wps.com/thumbnails/ems-pausing-during-database-deployment-lesson-learned/240351.png","ImageObject",442,249,{"name":88,"@type":89},"Caleb Sterling","Person",{"url":68,"name":91,"@type":92},"DocShare","Organization","application/pdf","2026-09-25","2026-09-11",true,{"@type":98,"interactionType":99,"userInteractionCount":76},"InteractionCounter",{"@type":100},"ViewAction",{"@type":102,"mainEntity":103},"FAQPage",[104,110,114],{"name":105,"@type":106,"acceptedAnswer":107},"What problem is described in the EMS database deployment incidents?","Question",{"text":108,"@type":109},"Entities experienced EMS outages during EMS database deployment, with the same EMS vendor involved across all cases. The outages paused key databases and led to loss of control center visibility.","Answer",{"name":111,"@type":106,"acceptedAnswer":112},"Why did the databases remain paused after promotion attempts?",{"text":113,"@type":109},"In different cases, critical processes disconnected, restarted, or the promotion process halted after pausing critical SCADA data. In each situation, monitoring and alarm/notification coverage was insufficient to detect the paused state promptly.",{"name":115,"@type":106,"acceptedAnswer":116},"What corrective actions were taken to restore visibility and operation?",{"text":117,"@type":109},"Entities contacted the vendor and took steps such as syncing SCADA databases, restarting FEP services, executing an FEP build, reverting to preupdated conditions, and unpausing the affected databases. Vendor enhancements were also pursued to reduce recurrence and speed recovery.","https://schema.org",{"og:url":78,"og:type":120,"og:title":59,"og:site_name":91,"og:description":61},"article",{"robots":122,"canonical":78},"index,follow",{"doc_id":124,"site_id":56},240351,1790189267,{"code":4,"msg":5,"data":127},{"doc_id":124,"user_id":128,"nickname":88,"user_avatar":129,"doc_module":9,"category_id":50,"category_name":51,"doc_title":59,"doc_description":61,"doc_content":130,"file_id":131,"file_url":132,"file_type":133,"file_size":134,"view_count":76,"is_deleted":4,"is_public":9,"is_downloadable":9,"audit_status":9,"page_count":76,"language":135,"language_code":57,"site_id":56,"html_lang":57,"table_of_contents":136,"faqs":137,"seo_title":138,"seo_description":61,"update_tm":139,"read_time":9},962084925290,"https://ap-avatar.wpscdn.com/davatar_085a072bc5b1113ac321206ff7593b45","Lesson Learned  \nEMS Pausing During Database Deployment  \nPrimary Interest Groups  \nTransmission Operators (TOPs)  \nTransmission Owners (TOs)  \nReliability Coordinators (RCs)  \nBalancing Authorities (BAs)  \nProblem Statement  \nSeveral entities have experienced energy management system (EMS) outages during EMS database deployment. All of the cases had the same EMS vendor.  \nDetails  \nCase 1  \nThe entity added new points for inter-control center communications protocol (ICCP) and supervisory control and data acquisition (SCADA) . The entity pushed changes to the production environment after a successful validation at the quality assurance system. During the database promotion, several databases (including the front end processor (FEP) and SCADA databases) were paused as expected. Due to issues encountered during the promotion, the databases were not unpaused at the completion of the effort, causing the loss of the control center visibility. The entity attempted to revert the changes, but it was unsuccessful and did not unpause the databases.  \nAfter a post-event investigation, it was discovered that a critical process that manages the database promotion had disconnected before the task was executed. The critical process connection was not readily monitored, and there was no obvious alarm or indication when it had disconnected.  \nCase 2  \nThe entity scheduled a planned database promotion, including changes to FEP and SCADA databases, from the quality assurance system to the production environment. When errors were reported during the database promotion, system staff manually terminated the effort midway. The termination left FEP and SCADA databases paused and caused the loss of the control center visibility. The entity attempted to revert the promotion, but the effort failed.  \nAfter a post-event investigation, it was discovered that the errors observed that caused staff to terminate the promotion were due to a critical process restarting on the production environment. While the process restarted successfully, the databases were left in a paused state due to the database promotion termination.  \nCase 3  \nThe entity executed a routine database promotion. After successfully updating four remote FEP databases,  \nthe promotion process, by design, paused a database that contained critical SCADA data. However, after the database was paused, the promotion process failed to execute the next step and stopped further tasks, leaving the databases paused. There was no obvious alarm or notification to alert that the database had been paused. As a result, the entity did not realize the databases were paused until operators found they could not change EMS screens. The entity attempted to revert the promotion; however, the attempt to revert failed. Swapping could not work because database promotions impact all production servers.  \nThe EMS vendor was not able to systematically reproduce the condition where the database failed to restart at that step in the data model promotion. However, they were able to confirm through a simulation that pausing the database will cause a loss of monitoring and control. The vendor is continuing to work on identifying a root cause.  \nCorrective Actions  \nIn each of these cases, upon noticing the degradation of the situational awareness, the entity contacted the appropriate personnel (e.g., RC and operations support staff) to assist with monitoring and help repair the situation.  \nCase 1  \nThe entity contacted and worked with the vendor for support and recovery. The first step to repairing the issue was to sync the SCADA databases. The second step was to restart an FEP service on the active server. Last, a FEP build was executed to regain sight and control of the SCADA system.  \nCase 2  \nThe entity contacted the vendor for emergency help. The vendor assistance was able to revert to the preupdated system condition and restored visibility. The entity had dispatched personnel to the backup facility to perform operatio","cbCaicfRoCF2tFiy","https://ap.wps.com/l/cbCaicfRoCF2tFiy","pdf",691153,"English","# Primary Interest Groups\n# Problem Statement\n# Details\n## Case 1\n## Case 2\n## Case 3\n# Corrective Actions\n## Case 1\n## Case 2\n## Case 3\n# Lessons Learned\n# NERC’s goal","[{\"question\":\"What problem is described in the EMS database deployment incidents?\",\"answer\":\"Entities experienced EMS outages during EMS database deployment, with the same EMS vendor involved across all cases. The outages paused key databases and led to loss of control center visibility.\"},{\"question\":\"Why did the databases remain paused after promotion attempts?\",\"answer\":\"In different cases, critical processes disconnected, restarted, or the promotion process halted after pausing critical SCADA data. In each situation, monitoring and alarm/notification coverage was insufficient to detect the paused state promptly.\"},{\"question\":\"What corrective actions were taken to restore visibility and operation?\",\"answer\":\"Entities contacted the vendor and took steps such as syncing SCADA databases, restarting FEP services, executing an FEP build, reverting to preupdated conditions, and unpausing the affected databases. Vendor enhancements were also pursued to reduce recurrence and speed recovery.\"}]","EMS Pausing During Database Deployment - Lesson Learned | PDF",1789160273]