Evaluating the Performance of Agreement Metrics in a Delphi Study on Chemical, Biological, Radiological and Nuclear Major Incidents Preparedness Using Classical and Machine Learning Approaches

Aug 4, 2026

Upload by Eden

Delphi studies in disaster medicine often lack consensus on how expert agreement metrics perform. This study evaluates agreement measures within a Delphi process for chemical, biological, radiological, and nuclear (CBRN) major-incident preparedness in the Middle East and North Africa. Forty international experts rated 133 items across ten CBRN Preparedness Assessment Tool themes on a 5-point Likert scale. Kendall’s W, intraclass correlation coefficient, and Cohen’s kappa were compared using statistical and machine-learning methods. Overall agreement reached 4.91 ± 0.71, and kappa proved the most sensitive metric, varying by themes such as medical protocols, logistics, and infrastructure, supporting future refinement.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages20

Reading Time50 min

AccessFree to download

Introduction

Delphi studies in disaster medicine

Expert consensus, benefits, and criticisms

Methods and analysis

Delphi process and assessment items

Agreement metrics and comparison approaches

Results and interpretation

Overall agreement and metric sensitivity

Variations across CBRN themes

Discussion and implications

Integrated statistical and machine learning value

Journal of Contingencies and Crisis Management ORIGINAL ARTICLE Evaluating the Performance of Agreement Metrics in a Delphi Study on Chemical, Biological, Radiological and Nuclear Major Incidents Preparedness Using Classical and Machine Learning Approaches Hassan Farhat1,2 | Alan M. Batt3,4 | Mariana Helou5,6 | Heejun Shin7,8,9, 10 | James Laughton2 | Carolyn Dumbeck11 | Arezoo Dehghani12, 13 | Fatemeh Rezaei13 | Nidaa Bajow14 | Luc Mortelmans15,16, 17 | Walid Abougalala18 | Roberto Mugavero19,20,21 | Gregory Ciottone10,22 | Guillaume Alinier2,23,24,25 | Mohamed Ben Dhiab1 1Faculty of Medicine “Ibn El Jazzar,”, University of Sousse, Sousse, Tunisia | 2Ambulance Service, Hamad Medical Corporation, Doha, Qatar | 3Queen's University, Kingston, Ontario, Canada | 4Monash University, Melbourne, Victoria, Australia | 5School of Medicine, Lebanese American University, Beirut, Lebanon | 6Lebanese American University‐Rizk Hospital, Beirut, Lebanon | 7Soonchunhyang Disaster Medicine Center, Bucheon, South Korea | 8Soonchunhyang University Bucheon Hospital, Bucheon, South Korea | 9Shin's Disaster Medicine Academy, Seoul, South Korea | 10Harvard Medical School, Harvard University, Cambridge, Massachusetts, USA | 11Department of Disaster Management, Alberta Health Services | 12Safety Promotion and Injury Prevention Research Center, Shahid Beheshti University of Medical Sciences, Tehran, Iran | 13Department of Health in Emergencies and Disasters, Health Management and Economic Research Center, Isfahan University of Medical Sciences, Isfahan, Iran | 14Security Forces Hospital, Riyadh, Saudi Arabia | 15European Society for Emergency Medicine, Belgium | 16Catholic University of Leuven, Belgium | 17Free University Brussels, Belgium | 18Hamad Medical Corporation, Doha, Qatar | 19Department of Electronic Engineering – DIE, University of Rome “Tor Vergata, Rome, Italy | 20Centre for Security Studies – CUFS, University of the Republic of San Marino, San Marino | 21Observatory on Security and CBRNe Defense – OSDIFE | 22Harvard T.H. Chan School of Public Health, USA | 23School of Health and Social Work, University of Hertfordshire, Hatfield, UK | 24Weill Cornell Medicine‐Qatar, Doha, Qatar | 25Faculty of Health and Life Sciences, Northumbria University, Newcastle upon Tyne, UK Correspondence: Hassan Farhat ([hassen.farhat@gmail.com](hassen.farhat@gmail.com)) Received: 9 December 2024 | Revised: 24 March 2025 | Accepted: 24 March 2025 Funding: The authors received no specific funding for this study. Keywords: agreement analysis | Delphi study | disaster medicine | expert's opinion | MENA ABSTRACT Delphi studies in disaster medicine lack consensus on expert agreement metrics. This study examined various metrics using a Delphi study on chemical, biological, radiological, and nuclear (CBRN) preparedness in the Middle East and North Africa region. Forty international disaster medicine experts evaluated 133 items across ten CBRN Preparedness Assessment Tool themes using a 5‐point Likert scale. Agreement was measured using Kendall's W, Intraclass Correlation Coefficient, and Cohen's Kappa. Statistical and machine learning techniques compared metric performance. The overall agreement mean score was 4.91 ± 0.71, with 89.21% average agreement. Kappa emerged as the most sensitive metric in statistical and machine learning analyses, with a feature importance score of 168.32 . The Kappa coefficient showed variations across CBRN PAT themes, including medical protocols, logistics, and infrastructure. The integrated statistical and machine learning approach provides a promising method for understanding expert consensus in disaster preparedness, with potential for future refinement by incorporating additional contextual factors. This is an open access article under the terms of the Creative Commons Attribution License, which permits use, distribution and reproduction in any medium, provided the original work is properly cited. ©

FAQs

What is the study evaluating in the Delphi process?

It evaluates how different agreement metrics perform when experts assess chemical, biological, radiological, and nuclear preparedness items in a Delphi study.

How were expert agreement metrics measured?

Agreement was measured using Kendall’s W, intraclass correlation coefficient, and Cohen’s kappa, then compared with statistical and machine-learning techniques.

Which agreement metric was most sensitive, and what did it show?

Cohen’s kappa was the most sensitive metric, showing variations across themes including medical protocols, logistics, and infrastructure.

Aug 4, 2026

Upload by Eden