Rethinking Software Empirical Studies with Structural Causal Models - Tutorial methodology and causal inference case study

Aug 26, 2026

Uploaded by Noah

Causal Inference reframes empirical software engineering by targeting causal relationships rather than relying solely on statistical association. The paper presents CausalSE, a framework that operationalizes Judea Pearl’s causal inference paradigm within software engineering using Structural Causal Models to mitigate confounding bias. A case study on the Galeras dataset with propensity score matching examines how prompt engineering affects code generation in GPT-3, showing that confounding-adjusted causal analysis can overturn associational findings. The methodology supports rigorous design, analysis, and interpretation of software experiments.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages22

Reading Time55 min

AccessFree to download

Introduction

Causal inference and mechanistic causality

Pearl’s do-calculus and the ladder of causation

Structural causal models and confounding/selection bias

arXiv :2605 .28482v1 [ cs . SE] 27 May 2026 Rethinking Software Empirical Studies with Structural Causal Models DANIEL RODRIGUEZ-CARDENAS∗ , AYA GARRYYEVA∗ , DAVID N. PALACIO∗ , ANTONIO MASTROPAOLO, and DENYS POSHYVANYK, William & Mary, USA Causal Inference offers a fundamental approach for advancing empirical software engineering (ESE) beyond traditional statistical association, enabling researchers to rigorously identify and quantify causal relationships in software experiments. This paper introduces CausalSE, a framework that operationalizes Judea Pearl’s causal inference paradigm in ESE context. The paper focuses on Structural Causal Models (SCMs) to address the limitations of classical statistical methods in mitigating confounding bias. Through a case study using the Galeras dataset and propensity score matching, we demonstrate how CausalSE disentangles the effect of prompt engineering strategies on code generation outcomes in a popular LLM (i.e., GPT-3) . The results reveal that while associational analyses can suggest improvements in certain interventions (e.g., more complex prompts), causal analysis often does not find a significant treatment effect, highlighting the risk of false positives when confounding is not addressed. By providing a tutorial-based methodology and a real-world case study, this work equips software researchers with practical tools to design, analyze, and interpret software experiments with methodological rigor, ultimately enabling more informed and actionable conclusions in both research and practice. CCS Concepts: • Mathematics of computing → Hypothesis testing and confidence interval computation; Probabilistic representations. Additional Key Words and Phrases: Causal Graphs, Empirical SE, Spurious Correlation ACM Reference Format: Daniel Rodriguez-Cardenas, Aya Garryyeva, David N. Palacio, Antonio Mastropaolo, and Denys Poshyvanyk. 2026. Rethinking Software Empirical Studies with Structural Causal Models . 1, 1 (May 2026), 22 pages. [https://doi.org/XXXXXXX.XXXXXXX](https://doi.org/XXXXXXX.XXXXXXX) 1 Introduction Scientific explanation and causality are deeply intertwined: for most philosophers of science, explaining a phenomenon is to identify what caused it. The debate traces back to Aristotle (322 BCE), who organized scientific inquiry around why-type questions, and to Hume (1775), who countered that we never observe causation directly; only the habitual conjunction of events. To this day, no single definition of causality has received universal acceptance, either philosophically or scientifically. Contemporary authors argue for Humean reduction. Cartwright [8–10] and Bunge [5–7] argue that causation must be grounded in mechanisms rather than statistical correlations1 . The two diverge on the details: Cartwright admits context-dependent causation in place of strict deterministic laws, while Bunge retains determinism but embeds it ∗ Authors contributed equally. 1Bunge defines a “mechanism” as a process in a concrete system whereby one state or event brings about another under laws of nature [7] . Authors’ Contact Information: Daniel Rodriguez-Cardenas, [dhrodriguezcar@wm.edu](dhrodriguezcar@wm.edu); Aya Garryyeva, [lgarryyeva@wm.edu](lgarryyeva@wm.edu); David N. Palacio, davidnad@ [microsoft.com](microsoft.com); Antonio Mastropaolo, [amastropaolo@wm.edu](amastropaolo@wm.edu); Denys Poshyvanyk, [dposhyvanyk@wm.edu](dposhyvanyk@wm.edu), William & Mary, Williamsburg, VA, USA. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, [requires

FAQs

What problem does the paper address in empirical software engineering studies?

It addresses limitations of classical statistical association methods, especially confounding bias, which can produce misleading conclusions about causal effects.

What is CausalSE and what does it operationalize?

CausalSE is a framework that operationalizes Judea Pearl’s causal inference paradigm for empirical software engineering using Structural Causal Models.

How does the case study evaluate the effect of prompt engineering on code generation?

It uses the Galeras dataset and propensity score matching to disentangle the treatment effect of prompt engineering strategies on GPT-3 code generation outcomes.

Why can associational analyses and causal analyses lead to different conclusions?

Associational analyses may indicate improvements even when confounding is present, while causal analyses—after accounting for confounding—may find no significant treatment effect, reducing false positives.

Aug 26, 2026

Uploaded by Noah