Multi-Metric Adaptive Experimental Design Under a Fixed Budget with Validation

Jul 16, 2026

Upload by Himbo

A/B tests in online experimentation struggle with statistical power when many candidate treatments are tested simultaneously, while adaptive experimental design (AED) alone cannot reliably infer key experiment statistics such as the average treatment effect (ATE), particularly under many metrics and heterogeneous variances. The paper introduces a fixed-budget multi-metric AED framework with two phases: adaptive exploration to select the best treatment, and validation via an A/B test to verify quality and infer statistics.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages48

Reading Time2 h 1 min

Introduction

Experimentation Framework

Challenges

Related Work

Contributions

Multi-Metric Adaptive Experimental Design Under a Fixed Budget with Validation Qining Zhang* University of Michigan [qiningz@umich.edu](qiningz@umich.edu) Tanner Fiez Amazon [tannfiez@amazon.com](tannfiez@amazon.com) Yi Liu Amazon [yiam@amazon.com](yiam@amazon.com) arXiv :2506 .03062v2 [ cs .LG] 10 Jul 2026 Wenyang Liu Amazon [lwenyang@amazon.com](lwenyang@amazon.com) Abstract A/B tests in online experiments face statistical power challenges when testing multiple candidates simultaneously, while adaptive experimental designs (AED) alone fall short in inferring experiment statistics such asthe average treatment effect, especially with many metrics (e.g., revenue, safety) and heterogeneous variances. This paper proposes a fixed-budget multi-metric AED framework with a two-phase structure: an adaptive exploration phase to identify the best treatment, and a validation phase with an A/B test to verify the treatment’s quality and infer statistics. We propose SHRVar, which generalizes sequential halving ( SH) with a novel relative-variance-based sampling and an elimination strategy built on reward z values. It achieves a provable error probability that decreases exponentially, where the exponent H3 generalizes the complexity measure for SH and SHVar with homogeneous and heterogeneous variances, respectively. Numerical experiments demonstrate its performance and robustness. 1 Introduction Randomized online experimental design, which aims to evaluate and compare the performance of different system versions, is a standard procedure to support statistically valid decisions in industrial developments, such as new webpage layouts, new services, and new software features. A/B test (or A/B/N test), where the designer assignsan experiment subject to either the control (current version) or one of the multiple treatments (new versions) with a fixed probability and then measures their response, has demonstrated empirical success (Kohavi et al., 2020) due to its easy-to-implement nature and the accuracy in inferring experimental statistics such as the average treatment effect (ATE) (Imbens, 2004) . However, A/B tests demonstrate poor statistical power when the number of treatments scales, which becomes insufficient in modern experiments where hundreds of new treatments maybe developed simultaneously from machine learning methods such as generative AIs. On the other hand, adaptive experimental design (AED) methods, such as best arm identification (BAI), where the probability of experiment subject assignment can be adaptively chosen, have received increasing popularity as an alternative to reduce the cost of experimentation. However, the adaptivity also prohibits designers from the merits of classic A/B tests, such as the accurate inference of ATE (Cook et al., 2023; Deep et al., 2023) . Experimentation Framework. To combine the advantages of both AED and A/B tests, specifically fast and efficient treatment selection in experimentation with accurate statistical inference, we propose and study an *The research was mainly conducted when Qining Zhang was a research scientist intern at Amazon. experimentation framework shown in Fig. 1. This framework consists of two phases, an exploration phase anda validation phase. In exploration, an AED method, such as a multi-armed bandit (MAB) algorithm, is employed to identify the “best” treatment, for example, a new layout of the shopping website preferred by customers. Then, in validation, the recommended treatment (new layout) and the control (old layout) are placed in an A/B test to verify their performance in multiple metrics and obtain experiment statistics such as the ATE. When the superior-than-control performance of the recommended treatment is validated under all metrics, the new layout will be deployed into production to replace the old website layout, and the cycle continues. Therefore, the exploration phase aims to identify a treatment most likely to pass the validat

FAQs

What problem does the paper address in multi-treatment A/B testing?

It addresses poor statistical power when testing many candidates simultaneously and the inability of standalone adaptive designs to accurately infer statistics like the ATE under multiple metrics and heterogeneous variances.

How is the proposed experimentation framework structured?

It uses two phases: an adaptive exploration phase to identify the best treatment, followed by a validation phase running an A/B test to verify treatment quality across multiple metrics and infer statistics.

What is SHRVar and what does it improve?

SHRVar generalizes sequential halving by using relative-variance-based sampling and an elimination strategy based on reward z values, providing an exponentially decreasing provable error probability and robust performance in experiments.

Jul 16, 2026

Upload by Himbo