K-ABENA K-Adaptive Backpropagation with Error-based N-exclusion Algorithm

Jul 16, 2026

Upload by Liam

K-ABENA introduces a selective gradient computation framework that lowers per-iteration training cost by excluding a fraction of low-loss (“minor”) observations from the backward pass. The canonical v3 design uses a defensive-mixture sampling strategy over the minor set combined with Horvitz–Thompson inverse-probability reweighting, providing a design-unbiased gradient estimator and a self-normalized practical variant with bias shrinking as O(1/m). It proves O(1/√T) non-convex SGD convergence with an explicit residual-bias term, and shows uncompensated loss-based selection prevents convergence under imbalance or noise.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages13

Reading Time33 min

Abstract

1 Introduction

Contributions

arXiv :2607 .05903v 1 [ cs .LG] 7 Jul 2026 K-ABENA: K-Adaptive Backpropagation with Error-based N-exclusion Algorithm Compensated Loss-Based Sample Exclusion with Unbiased Gradient Estimation Jean-Fran¸cois Bonbhel NeuroSoft IA, Qu´ebec City, Canada | YekoElite University, Brazzaville, Republic of Congo UN AI Governance Expert Network (UN PNAI)—Member since 2021 [bonbhel@yekoelite.com](bonbhel@yekoelite.com) July 2026 Abstract We present K-ABENA (K-Adaptive Backpropagation with Error-based N-exclusion Algorithm), a selective gradient computation framework that reduces per-iteration training cost by excluding a fraction of low-loss (“minor”) observations from the backward pass. Its canonical form (v3) combines a defensive-mixture sampling design over the minor set with Horvitz–Thompson inverse-probability reweighting, yielding a design-unbiased Horvitz–Thompson gradient estimator (Lemma 2) and whose self-normalized practical variant carries a bias of order O(1/m) with an explicit constant (Lemma 3) . We prove an O(1/ √T ) non-convex convergence guarantee for SGD under the estimator, with an additive term that quantifies the residual bias (Theorem 1) . We further prove that uncompensated loss-based selection—a family that includes OHEM, SBP, and the two earlier K-ABENA variants—admits no stationary point at any minimizer where its selection bias is bounded away from zero (Proposition 2), and we quantify this failure empirically: at 0.17% class imbalance, uncompensated variants reach test AUC 0.53–0.62 versus 0.9998 for full-batch SGD, while the compensated estimator attains 0 .9991 at identical 28 .4% compute savings. On real datasets (Breast Cancer, Digits, Wine, Diabetes) the compensated estimator is statistically indistinguishable from full-batch SGD (paired permutation tests, p ≥ 0.5; Section 7) while saving 28–54% of per-epoch gradient computation. A biased “regularized mode”(the earlier half-domain variant) is retained as an option with a proven exact bias decomposition (Lemma 5) and quantified contraindications: it collapses to 0.386 accuracy under 40% label noise (baseline: 0.832) and to 0.53 AUC under extreme imbalance. Every advantage and every limitation reported in this paper is either proved or measured; all experiments are CPU-scale (NumPy/scikit-learn) and their scope is stated explicitly. 1 Introduction In large-scale empirical risk minimization, a substantial fraction of per-iteration computation is spent on observations the model has already learned: their per-sample losses are small, their gradients are small, and their marginal contribution to the descent direction is limited. Selectivebackpropagation methods exploit this observation by skipping the backward pass for low-loss samples [6, 12], but they share a structural defect: the retained subset is correlated with the loss, so the resulting gradient is a biased estimator of the full-batch gradient. In benign regimes the bias is small relative to the signal and these methods work well; we show in Section 5 that in adverse regimes — extreme class imbalance, heavy label noise—the bias does not merely degrade performance but structurally prevents convergence to the minimizer, and we prove this as Proposition 2. This paper develops K-ABENA, whose canonical estimator (referred to as v3) resolves the defect using a century-old idea from survey sampling [5]: any sampling design with known, strictly positive inclusion probabilities admits an unbiased estimator of a population total via inverse-probability weighting. K-ABENA v3 samples retained minors from the entire minor set under a defensive mixture design [10] and reweights them accordingly. The result occupies a design point that, to our knowledge, none of the established selective or reweighting methods occupies: per-iteration compute reduction with an (exactly or near-) unbiased gradient. Hardselection methods (OHEM [12], SBP [6]) save compute but are biased; soft-reweighting methods (F

FAQs

What problem does K-ABENA address in selective backpropagation?

Selective backpropagation skips the backward pass for low-loss samples to save compute, but the retained subset correlates with loss, creating a biased gradient. K-ABENA targets this bias while still reducing backward computation.

How does the canonical K-ABENA (v3) reduce gradient bias?

It samples retained “minor” observations using a defensive-mixture design and applies Horvitz–Thompson inverse-probability reweighting. This yields a design-unbiased estimator, and a self-normalized variant with bias that decreases on the order of O(1/m).

What does the paper show about uncompensated loss-based selection?

Uncompensated selection methods cannot have a stationary point at a minimizer when the selection bias remains bounded away from zero. Experiments indicate large performance degradation under class imbalance, while the compensated estimator restores near full-batch performance.

Jul 16, 2026

Upload by Liam