Can Variation in Subgroups’ Average Treatment Eﬀects Explain Treatment Eﬀect Heterogeneity? Evidence from a Social Experiment - Draft paper with appendix figures

Sep 4, 2026

Uploaded by Miles

This paper evaluates whether welfare reform affects earnings solely through mean impacts that are constant within subgroups yet vary across them. The analysis targets a common practice in treatment-effect heterogeneity research that restricts attention to mean impacts varying only by subgroup. Using a novel simulation approach for treated-group earnings under the constant mean-impacts-within-subgroup model, the study finds poor capture of Jobs First treatment heterogeneity in Connecticut, using quantile treatment effects. It also shows that excluding within-group heterogeneity would conceal results consistent with core labor supply theory predictions.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages35

Reading Time1 h 28 min

AccessFree to download

Abstract

1 Introduction

Background on quantile treatment effects

Conventional mean-impact-by-subgroup approach

Proposed simulated earnings under the null model

Can Variation in Subgroups’ Average Treatment Eﬀects Explain Treatment Eﬀect Heterogeneity? Evidence from a Social Experiment Marianne P. Bitler University of California, Irvine and NBER Jonah B. Gelbach University of Pennsylvania Law School Hilary W. Hoynes University of California, Berkeley and NBER 1 This version: April 2014 1 Correspondence to Hoynes at Richard & Rhoda Goldman School of Public Policy, UC Berkeley, 2607 Hearst Avenue, Berkeley, CA 94720-7320, phone (510) 642-1166, fax (510) 643-9657, [or](or hoynes@berkeley.edu)[ h](or hoynes@berkeley.edu)[oynes@berkeley.edu](or hoynes@berkeley.edu); Gelbach at [j](jgelbach@law.upenn.edu)[gelbach@law.upenn.edu](jgelbach@law.upenn.edu); or Bitler at [m](mbitler@uci.edu. The)[bitler@uci.edu. The](mbitler@uci.edu. The) data used in this paper are derived from data ﬁles made available to researchers by MDRC. The authors remain solely responsible for how the data have been used or interpreted. We are very grateful to MDRC for providing the public access to the experimental data used here. We would also like to thank Alberto Abadie, Michael Anderson, Joe Altonji, Richard Blundell, Mike Boozer, David Brownstone, Moshe Buchinsky, Raj Chetty, Julie Cullen, Joe Cummins, Peng Ding, Avi Feller, David Green, Jeﬀ Grogger, Jon Guryan, John Ham, Pat Kline, Thomas Lemieux, Bruce Meyer, Luke Miratrix, Robert Moﬃtt, Enrico Moretti, Giuseppe Ragusa, Shu Shen, Jeﬀ Smith, Melissa Tartari, and Rob Valetta for helpful conversations, as well as seminar participants at the IRP Summer Research Workshop, the SOLE meetings, the IZA-SPEAC conference, the Harris School, UBC, UC Davis, UC Irvine, UCL, UCLA, UCSD, UCSC, UCSB, the San Francisco Federal Reserve Bank, Tinbergen Institute, Toronto, and Yale University. Abstract In this paper, we assess whether welfare reform aﬀects earnings only through mean impacts that are constant within but vary across subgroups. This is important because researchers interested in treatment eﬀect heterogeneity typically restrict their attention to estimating mean impacts that are only allowed to vary across subgroups. Using a novel approach to simulating treatment group earnings under the constant mean-impacts within subgroup model, we ﬁnd that this model does a poor job of capturing the treatment eﬀect heterogeneity for Connecticut’s Jobs First welfare reform experiment using quantile treatment eﬀects. Notably, ignoring within-group heterogeneity would lead one to miss evidence that the Jobs First experiment’s eﬀects are consistent with central predictions of basic labor supply theory. 1 Introduction In previous work we estimated quantile treatment eﬀects using data from a randomized experiment to evaluate the labor supply impact of Connecticut’s welfare reform program Jobs First (Bitler, Gelbach & Hoynes (2006)) . In that context, labor supply theory predicts that the reform should cause heterogeneous treatment eﬀects across the earnings distribution. The data revealed a pattern consistent with these predictions, which, roughly speaking, are that there should be mass points at zero earnings in both the treated and control distributions, positive earnings eﬀects in the middle of the earnings distribution, and negative earnings eﬀects at the top of the earnings distribution. We found exactly this pattern of results, as Figure 3 of our earlier paper shows, using estimated quantile treatment eﬀects (QTE) of Jobs First on the earnings distribution. Moreover, the range of QTE was quite broad, from-$300 to $500, far above the mean impact of $82 . Our QTE-based approach to measuring treatment eﬀect heterogeneity diﬀers from the conventional one. One common approach involves estimating mean treatment eﬀects but allowing the treatment eﬀects to vary across subgroups based on demographic or other covariates. One then evaluates whether the subgroup-speciﬁc diﬀerences in treatment eﬀects appear to vary importantly (for a review within the welfare reform

FAQs

What question does the paper investigate about welfare reform effects?

It assesses whether welfare reform changes earnings only through subgroup mean impacts that are constant within subgroups but vary across subgroups.

How does the paper’s approach to treatment-effect heterogeneity differ from the conventional one?

The conventional approach estimates mean treatment effects that vary across subgroups using covariates. This paper instead simulates treated earnings under a “mean-impacts-only” within-subgroup model and evaluates how well it reproduces heterogeneity using quantile treatment effects.

What do the findings say about the Jobs First experiment?

The constant mean-impacts-within-subgroup model performs poorly at capturing treatment-effect heterogeneity for Connecticut’s Jobs First experiment, and ignoring within-group heterogeneity would miss evidence aligned with basic labor supply theory predictions.

Sep 4, 2026

Uploaded by Miles