Learning from Noise Effective-Rank Collapse and Out-of-Distribution Rejection in Restricted Boltzmann Machines

Jul 16, 2026

Upload by Levi

Learning from Noise: Effective-Rank Collapse and Out-of-Distribution Rejection in Restricted Boltzmann Machines studies why RBM-based classifiers fail to reject out-of-distribution (OOD) inputs, even when in-distribution classification is accurate. The analysis focuses on the induced visible–visible interaction J = WWT, contrasting conventional training against a protocol that adds random binary images labeled as rejection. Auxiliary training collapses J’s effective rank by depleting bulk-like spectral modes and concentrating weight into dominant directions, reshaping both the interaction spectrum and the free-energy landscape. The result rejects structured OOD datasets while preserving MNIST performance.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages11

Reading Time28 min

Introduction

Energy-based view of RBM classification

OOD rejection failure mode

Auxiliary rejection-state training

Effective interactions via J = WWT

Learning from Noise: Effective-Rank Collapse and Out-of-Distribution Rejection in Restricted Boltzmann Machines Oshada Rathnayake∗ and Nikhil Shukla University of Virginia, Charlottesville, VA, 22903, USA Restricted Boltzmann machines (RBMs) represent data by shaping an energy landscape over visible and hidden configurations, but their discriminative use is fragile under out-of-distribution (OOD) inputs: samples outside the training distribution can be absorbed into one of the learned class basins rather than rejected. Here, we analyze this failure mode through the spectrum of the induced visible–visible interaction J = WW T , where W is the visible–hidden weight matrix. Relative to a Marchenko–Pastur random-matrix reference, conventional training spreads spectral weight into many weak, bulk-compatible directions, increasing the effective rank of J. When auxiliary random binary images are assigned to a rejection label during training, the learned interaction undergoes effective-rank collapse: weak bulk-like modes are depleted, spectral weight concentrates into fewer dominant eigendirections, and the effective rank of J approaches that of the empirical data covariance matrix. The resulting RBM rejects structured OOD image datasets while preserving MNIST classification accuracy, showing that random auxiliary exposure can reshape both the interaction spectrum and the free-energy landscape of an energy-based classifier. arXiv :2607 . 10506v 1 [ cs .LG] 11 Jul 2026 I. INTRODUCTION Restricted Boltzmann machines (RBMs) [1] are energy-based probabilistic models [2] in which learning corresponds to shaping an energy landscape over visible and hidden configurations. After training, configurations consistent with the data distribution are assigned low free-energy, whereas statistically incompatible configurations are expected to lie at higher free-energy [3, 4] . This representation allows RBMs to capture latent statistical structure in data and has supported their use in both generative modeling and discriminative inference [5–7] . In this work, we focus on the latter setting. As shown in Fig. 1(a), a conventionally trained RBM can achieve high classification accuracy on in-distribution data; similar results have been shown in prior work [8] as well. However, the same model exhibits a persistent limitation when evaluated on out-of-distribution (OOD) inputs [9– 11] . Since the classifier selects the lowest-energy label among the learned alternatives, an input far from the training distribution can still be absorbed into one of the in-distribution class basins rather than “rejected”. In the baseline model considered here, this behavior leads to vanishing OOD rejection accuracy, as shown in Fig. 1(c) . This failure mode exposes a structural limitation of the learned energy landscape: accurate classification within the training distribution does not, by itself, guarantee the formation of a separate rejection region for inputs outside that distribution. A direct way to introduce a rejection state is to augment the training set with auxiliary OOD samples [12] . In this work, these auxiliary samples are chosen to be random binary images, with each pixel independently drawn from a Bernoulli distribution with probability 1/2, and are assigned to an additional rejection label. As shown in ∗ [fcu8ss@virginia.edu](fcu8ss@virginia.edu) FIG. 1: Comparison of the digit classification and outof-distribution (OOD) rejection performance of an RBM trained exclusively on MNIST [13] and an RBM trained on MNIST + random binary images. Panels (a) and (b) show the digit classification accuracy on the MNIST test set, demonstrating comparable in-distribution performance for both models. Panels (c) and (d) show the OOD rejection performance on the KMNIST dataset [14] . Unlike the conventional RBM, the auxiliary-trained RBM successfully assigns the majority of OOD samples to the rejection class while preserving its digit classification perfor

FAQs

Why do conventionally trained RBM classifiers often fail at rejecting out-of-distribution inputs?

Because the classifier assigns the lowest free-energy label among learned alternatives, an OOD input can be absorbed into an in-distribution class basin rather than being assigned to a rejection state, leading to vanishing OOD rejection accuracy.

What role does the induced interaction J = WWT play in the analysis?

Since RBMs have no direct visible–visible couplings, correlations are mediated through visible–hidden weights. The induced interaction J = WWT provides a compact spectral diagnostic of how many and how strongly hidden units represent directions in visible space.

How does training with random auxiliary binary images change the spectrum of J and the rejection behavior?

Assigning random binary images to a rejection label suppresses expansion into Marchenko–Pastur bulk-like directions, producing an effective-rank collapse. Spectral weight concentrates into fewer dominant eigendirections, enabling the RBM to assign structured OOD inputs to the rejection class while keeping MNIST classification accuracy.

Jul 16, 2026

Upload by Levi