Statistical Adversaries: Natural Backdoor-like Features in Vision Datasets

Jul 16, 2026

Upload by Ezra

Model-specific adversarial attacks have been studied extensively, yet many rely on malicious insertion or victim-model optimization. This paper investigates a different failure mode: naturally occurring statistical signals in vision datasets that can function like backdoor-like triggers. Using ImageNet, it identifies label-linked patterns, applies statistical controls to remove random correlations, and shows that the resulting signals predictably shift model outputs. The effects are more targeted than generic corruption and transfer across CNN and transformer architectures.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages15

Reading Time38 min

Abstract

Introduction

Related Works

Adversarial Attacks

Statistical Adversaries: Natural Backdoor-like Features in Vision Datasets Paul K. Mandal 1 ,2 ,3 Pavan Reddy4 Tristan Malaty´nski5 [paul@research.neurint.ai](paul@research.neurint.ai) [pavan.reddy@gwmail.gwu.edu](pavan.reddy@gwmail.gwu.edu) [tristan@agh.edu.pl](tristan@agh.edu.pl) arXiv :2607 .055 16v 1 [ cs .CV] 6 Jul 2026 Abstract Model-specific adversarial attacks have been extensively studied. We study a different failure mode: naturally occurring statistical signals in vision data that can behave like backdoor-like triggers without being maliciously inserted. We call these signals statistical adversaries. We analyse Imagenet to find patterns that are strongly linked to certain labels. We then use statistical controls to remove random correlations from our candidate signals. Finally, we demonstrate that these signals directly and predictably alter model predictions. These statistical adversaries are more targeted than generic corruptions and transfer across different model architectures. This suggests that some vulnerabilities are driven by dataset structure and distribution rather than a single model’s idiosyncrasies. We conclude that ordinary datasets can contain exploitable adversarial surfaces even in the absence of poisoning, and suggest that dataset audits should treat spurious structure not only as a source of bias or interpretability failure, but also as a latent attack surface for vision models. 1. Introduction Although modern vision models achieve strong benchmark performance, these metrics do not guarantee that the features they use are semantically meaningful. Models often use spurious patterns for predictions; these models perform well on benchmarks, but fail to transfer over to real-world scenarios [4] . Ilyas et al. [10] further argues that adversarial examples are features learned from these poorly generalized, spurious patterns. Additionally, both natural and model audits have shown that ImageNet contains harmful spurious features, class-associated frequency shortcuts, and 1Neurint, LLC, Baton Rouge, LA, USA. 2U.S. Army Cyber Corps, U.S. Army Reserve, USA. 3Northwestern State University of Louisiana, Natchitoches, LA, USA. 4Automata, Arlington, VA, USA. 5AGH University of Krakow, Krakow, Poland. Correspondence to Paul K. Mandal: [paul@research.neurint.ai](paul@research.neurint.ai) systematic failure cases [8, 20, 35] . These findings suggest that ordinary, unpoisoned datasets may contain statistical structure that can be leveraged to cause model failures. Moosavi-Dezfooli et al. [15] demonstrated that a single adversarial direction can affect many images and transfer across different model architectures. Other work also shows that incorporating information about a target-class distribution can improve cross-model attacks [19] . These adversarial directions, however, are generally obtained through optimization against a single victim or surrogate model. Similarly, frequency-domain attacks use structured frequency components to efficiently search for adversarial perturbations [7] . These adversarial directions are identified through responses from the attacked model. Informationgeometric attacks similarly derive adversarial directions from Fisher geometry from a trained neural network [40] . Therefore, while prior work established that numerous different types of attack directions exist, these directions had to be identified through model-mediated signals. Our paper seeks to answer what these prior papers do not: can target-specific failure directions can instead be constructed from the statistics of the original source dataset, without optimizing a victim or surrogate attack objective? Establishing these directions would connect dataset-level statistics to model sensitivities that are shared across multiple different architectures. We refer to these data source derived directions that induce target-specific failures as statistical adversaries. Our study is focused on four

FAQs

What are “statistical adversaries” in this work?

They are directional perturbations derived only from the training data’s class-conditional statistics that can induce target-specific failures without being maliciously inserted.

How do the authors analyze ImageNet for these signals?

They search for patterns strongly linked to certain labels, then use statistical controls to remove random correlations from candidate signals before evaluation.

Do these dataset-derived signals transfer across models?

Yes. The paper reports that selected source-statistical directions transfer across different model architectures, producing targeted false-positive inflation, rank movement, and top-k entry rather than consistent top-1 takeover.

Jul 16, 2026

Upload by Ezra