Do Checklists Make a Difference - A Natural Experiment from Food Safety Enforcement

Sep 18, 2026

Uploaded by Kyle

Explores whether checklists improve law and public-policy administration using a natural experiment in Seattle and King County after a 2005 health-code revision. Half of inspection items were converted to checklist format while others stayed on free-form recall. Qualitative review identifies unchanged code items across revisions, then difference-in-differences estimates checklist effects in 95,000+ inspections. Results show no detectable impact on inspector behavior, while escalating violations from “noncritical” to “critical” meaningfully affects outcomes.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages36

Reading Time1 h 31 min

AccessFree to download

I. Introduction

Background and claims about checklists in law

Critiques and limits of checklist approaches

Journal of Empirical Legal Studies Volume 15, Issue 2, 00–00, June 2018 Do Checklists Make a Difference? A Natural Experiment from Food Safety Enforcement Daniel E. Ho, Sam Sherman, and Phil Wyman* Inspired by Atul Gawande’s bestselling Checklist Manifesto, many commentators have called for checklists to solve complex problems in law and public policy. We study a unique natural experiment to provide the first systematic evidence of checklists in law. In 2005, the Public Health Department of Seattle and King County revised its health code, subjecting half of inspection items to a checklist, with others remaining on a free-form recall basis. Through in-depth qualitative analysis, we identify the subset of code items that remained substantively identical across revisions, and then apply difference-in-differences to isolate the checklist effect in more than 95,000 inspections from 2001--2009 . Contrary to scholarly and popular claims that checklists can improve the administration of law, the checklist has no detectable effect on inspector behavior. Making a violation more salient by elevating it from “noncritical” to “critical” status, however, has a pronounced effect. The benefits of checklists alone are considerably overstated. I. Introduction Atul Gawande’s bestselling Checklist Manifesto argued that dramatic improvements in complex decisions can be achieved by cheap and easy-to-use checklists (Gawande 2009) . Generalizing from the evident success in surgery, Gawande claimed that checklists—simple, enumerated lists of information or steps—could improve complex decisions in “virtually any endeavor” (2009:158) . While Gawande acknowledged potential limitations, others quickly latched on: “Checklists are hot [and have] captured the imagination of the media and have inspired publication of a manifesto on their *Address correspondence to Daniel E. Ho, William Benjamin Scott and Luna M. Scott Professor of Law, Stanford Law School; Professor (by courtesy) of Political Science; Senior Fellow, Stanford Institute for Economic and Policy Research; 559 Nathan Abbott Way, Stanford, CA 94305; email: [dho@law.stanford.edu. Sherman](dho@law.stanford.edu. Sherman) is J.D. Candidate, Stanford Law School; Wyman was Health and Environmental Investigator, Public Health—Seattle and King County and is Environmental Health Specialist, Snohomish Health District. We thank Mark Bossart for help securing data, Janet Anderberg for insights into the health code revision, Becky Elias, Ki Straughn, and Dalila Zelkanovic for help in developing and carrying out a simplified inspection model, Claire Greenberg and Aubrey Jones for research assistance, and Becky Elias, Atul Gawande, Sandy Handan-Nader, Tim Lytton, Oluchi Mbonu, Bill Simon, David Studdert, and two anonymous referees for comments. 2 Ho et al. power in managing complexity” (Davidoff 2010:206) .1 Or as the Times put it, checklists are “a classic magic bullet” (Aaronovitch 2010) . Fueled in part by Gawande’s manifesto, scholars have, in turn, advocated for greater adoption of checklists in law and public policy. Richard Thaler and Cass Sunstein consider the checklist as part of choice architecture (Thaler et al. 2012:432) . Bill Simon (2012:394) urged “[p]rotocols of the sort that Gawande developed,” suggesting provocatively that checklists might have averted major ﬁnancial and accounting scandals, such as options backdating and Enron’s collapse. In criminal law, scholars have proposed checklists to help police and prosecutors comply with Brady obligations (Findley 2010; Grifﬁn 2011), to improve defense lawyering (Bibas 2011), and to help judges weigh testimony (Levine 2016) . A Federal Courts Study Committee recommended a legislative drafting checklist (Maggs 1992) and checklists have been proposed to improve election administration (Douglas 2015), safety regulation (Hale & Borys 2013), and education (Institute of Education Sciences & National Center for Education Statistics, D

FAQs

What natural experiment is used to study checklists in law?

The study uses a 2005 Seattle and King County health-code revision in which half of inspection items were assigned to a checklist format while others remained free-form recall.

How do the authors estimate the checklist effect?

They use in-depth qualitative analysis to identify code items that stayed substantively identical across revisions, then apply difference-in-differences to isolate the checklist effect.

What do the results show about checklist effects on inspector behavior?

The checklist has no detectable effect on inspector behavior, though making violations more salient by elevating them from “noncritical” to “critical” has a pronounced effect.

Sep 18, 2026

Uploaded by Kyle