LEAVE-ONE-OUT DISTINGUISHABILITY IN MACHINE LEARNING - Research framework for data memorization and privacy leakage

Aug 2, 2026

An analytical framework quantifies how a machine learning algorithm’s output distribution shifts when a small number of data points are included in (or removed from) its training set, formalized as leave-one-out distinguishability (LOOD). LOOD is positioned as a tool for measuring data memorization and information leakage, refining empirical metrics of privacy risk. Gaussian processes model algorithm randomness, and extensive experiments validate LOOD using membership inference attacks, identify high-leakage causes, and design optimal leave-one-out queries to enable training-data reconstruction.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages37

Reading Time1 h 33 min

AccessFree to download

Abstract

Introduction

Influence, memorization, and leakage

Leave-one-out distinguishability (LOOD)

Analytical method

Gaussian process modeling of output distributions

Empirical validation and experiments

Membership inference attacks and correlation tests

Mean-distance LOOD vs leave-one-out retraining

Theoretical explanations and algorithms

Most-influenced queries and reconstruction

Maximization behavior under NNGP and SGD

arXiv :2309 . 17310v4 [ cs .LG] 17 Apr 2024 LEAVE-ONE-OUT DISTINGUISHABILITY IN MACHINE LEARNING Jiayuan Ye†, Anastasia Borovykh‡, Soufiane Hayou†, and Reza Shokri† ∗† National University of Singapore,‡ Imperial College London ABSTRACT We introduce an analytical framework to quantify the changes in a machine learning algorithm’s output distribution following the inclusion of a few data points in its training set, a notion we define as leave-one-out distinguishability (LOOD) . This is key to measuring data memorization and information leakage as well asthe influence of training data points in machine learning. We illustrate how our method broadens and refines existing empirical measures of memorization and privacy risks associated with training data. We use Gaussian processes to model the randomness of machine learning algorithms, and validate LOOD with extensive empirical analysis of leakage using membership inference attacks. Our analytical framework enables us to investigate the causes of leakage and where the leakage is high. For example, we analyze the influence of activation functions, on data memorization. Additionally, our method allows us to identify queries that disclose the most information about the training data in the leave-one-out setting. We illustrate how optimal queries can be used for accurate reconstruction of training data.1 1 INTRODUCTION A key question in interpreting a model involves identifying which members of the training set have influenced the model’s predictions for a particular query (Koh & Liang, 2017; Koh et al., 2019; Pruthi et al., 2020) . Our goal is to understand the impact of the presence of a specific data point in the training set of a model on its predictions. This becomes important, for example, when model’s prediction on a data point is primarily due to its own presence (or that of points similar to it) in the training set-a phenomenon referred to as memorization (Feldman, 2020) and can cause leakage of sensitive information about the training data. This enables an adversary to infer which data points were included in the training set (using membership inference attacks), by observing the model’s predictions (Shokri et al., 2017; Zarifzadeh et al., 2023) . One way to measure such influence, memorization, and information leakage, and thus addressing related questions, is by (re)training models with various combinations of data points. This empirical approach, however, is computationally expensive (Feldman & Zhang, 2020), and does not enable efficient analysis of, for example, what data points are influential, which queries they influence the most, and what properties of model (architectures) can impact their influence. These limitations demand an efficient and constructive modeling approach to influence and leakage analysis. This is the focus of the current paper. We generalize the above-mentioned closely-related concepts (Section 2), and unify them as the statistical divergence between the output distribution of models, trained using (stochastic) machine learning algorithms, when one or more data points are added to (or removed from) the training set. We refer to this measure as leave-one-out distinguishability (LOOD) . A larger LOOD of a training data S in relation to query data Q suggests a larger potential for information leakage about S, when an adversary observes the prediction of models trained on S and queried at Q. As a special variant of LOOD, if we measure the gap between the mean of prediction distributions over leave-one-out models on a query, the resultant mean distance LOOD recovers existing definitions for influence (function) (Steinhardt et al., 2017; Koh & Liang, 2017; Koh et al., 2019) and memorization (selfinfluence) (Feldman & Zhang, 2020; Feldman, 2020) (i.e., a greater mean distance LOOD suggests a larger influence of training data S on query data Q) . ∗Authors AB, SH and RS are ordered alphabetically. 1The code is available at [https://github.

FAQs

What does leave-one-out distinguishability (LOOD) measure in machine learning?

LOOD measures how the output distribution of a model changes when specific data points are added to or removed from the training set in a leave-one-out setting.

How is LOOD used to assess privacy risks?

A larger LOOD suggests greater potential information leakage when an adversary observes model predictions on query data. The paper validates this link using membership inference attacks in leave-one-out experiments.

Why are Gaussian processes used in the proposed framework?

Gaussian processes are used to model the randomness of machine learning outputs under querying, enabling accurate and efficient estimation of LOOD without repeated retraining.

Aug 2, 2026

Upload by Ava Thompson