Evaluating Dynamic Conditional Quantile Treatment Effects with Applications in Ridesharing

Sep 4, 2026

Online experiments (A/B testing) are widely used by technology firms to compare new policies against existing ones, yet most methods target average treatment effects. For skewed and heavy-tailed outcomes, quantile treatment effects offer a richer distributional view, but dynamic conditional QTE estimation remains challenging when data reflect sequential decisions over time and space. This paper develops a formal framework for dynamic conditional QTE under conditional independence assumptions, proposes VCDP models, and designs a dynamic CQTE test, extending to spatiotemporal dependent experiments and demonstrating results on real ride-sourcing datasets with theory and simulations.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages57

Reading Time2 h 24 min

AccessFree to download

Introduction

Problem setting and motivation

Key challenges: dynamic QTE and curse of horizon

Contributions and proposed framework

arXiv :2305 . 10187v1 [ stat .ME] 17 May 2023 Evaluating Dynamic Conditional Quantile Treatment E􀀋ects with Applications in Ridesharing Ting Li􀀃1, Chengchun Shi􀀃2, Zhaohua Lu3 , Yi Li3 , and Hongtu Zhu4 ∗ 1 Shanghai University of Finance and Economics 2 London School of Economics and Political Science 3 DiDi Chuxing 4 University of North Carolina at Chapel Hill Abstract Many modern tech companies, such as Google, Uber, and Didi, utilize online experiments (also known as A/B testing) to evaluate new policies against existing ones. While most studies concentrate on average treatment e􀀋ects, situations with skewed and heavy-tailed outcome distributions may bene􀀌t from alternative criteria, such as quantiles. However, assessing dynamic quantile treatment e􀀋ects (QTE) remains a challenge, particularly when dealing with data from ride-sourcing platforms that involve sequential decision-making across time and space. In this paper, we establish a formal framework to calculate QTE conditional on characteristics independent of the treatment. Under speci􀀌c model assumptions, we demonstrate that the dynamic conditional QTE (CQTE) equals the sum of individual CQTEs across time, eventhough the conditional quantile of cumulative rewards may not necessarily equate to the sum of conditional quantiles of individual rewards. This crucial insight signi􀀌cantly streamlines the estimation and inference processes for our target causal estimand. We then introduce two varying coe􀀎cient decision process (VCDP) models and devise an innovative method to test the dynamic CQTE. Moreover, we expand our approach to accommodate data from spatiotemporal dependent experiments and examine both conditional quantile direct and indirect e􀀋ects. To showcase the practical utility of our method, we apply it to three real-world datasets from a ride-sourcing platform. Theoretical 􀀌ndings and comprehensive simulation studies further substantiate our proposal. Keywords: A/B testing, policy evaluation, quantile treatment e􀀋ect, ridesourcing platform, spatialtemporal experiments, varying coe􀀎cient models. ∗ The 􀀌rst two authors contribute equally to this paper. Address for correspondence: Hongtu Zhu, Ph.D. , [E-mail: htzhu@email.unc.edu. This work was partially](E-mail: htzhu@email.unc.edu. This work was partially) 􀀌nished when Dr Zhu worked at Didi Chuxing. The content is solely the responsibility of the authors and does not necessarily represent the o􀀎cial views of anyother funding agency. 1 Introduction Online experiments, often referred to as A/B testing in computer science literature, are widely utilized by technology companies (e.g., Google, Net􀀍ix, Microsoft) to assess thee􀀋ectiveness of new products or policies in comparison to existing ones. These companies have developed in-house A/B testing platforms for evaluating treatment e􀀋ects and providing valuable experimental insights. Take ridesourcing platforms like Uber, Lyft, and Didi as examples. These platforms operate within intricate spatiotemporal ecosystems, dynamically matching passengers with drivers (see, for instance, Wang and Yang, 2019; Qin et al. , 2020; Zhou et al., 2021) . They implement online experiments to explore various order dispatch policies and customer recommendation initiatives. These innovative products hold the potential to enhance passenger engagement and satisfaction, diminish pickup waiting times, and boost driver earnings, ultimately leading to a more e􀀎cient and user-friendly transportation system. In this study, we address the fundamental question of how to evaluate the di􀀋erence between the quantile return of a new product (treatment) and that of an existing one. Although the average treatment e􀀋ect (ATE) is widely used in the literature to quantify the di􀀋erence between two policies (Imbens and Rubin, 2015; Wang and Tchetgen Tchetgen, 2018; Kong et al., 2022), it only considers the average e􀀋ect and does not account for variability around the expectation. I

FAQs

What does the paper propose for evaluating dynamic conditional quantile treatment effects?

It establishes a formal framework to compute QTE conditional on characteristics independent of treatment, and under specific model assumptions it shows how dynamic conditional QTE can be expressed as sums of individual CQTEs over time.

Why is dynamic QTE estimation difficult in ride-sourcing experiments?

Because treatments are assigned sequentially across time and locations, existing methods can suffer from the curse of horizon, and the quantile of cumulative outcomes does not generally decompose into sums of individual quantiles.

How does the paper validate its method?

It introduces two varying coefficient decision process models, develops a testing method for dynamic CQTE, extends the approach to spatiotemporal dependent experiments, and applies it to three real ride-sourcing datasets with theoretical results and simulation studies.

Sep 4, 2026

Uploaded by Felix Montgomery