A Structural Interpretation of GELU and Threshold-Transmission Activations via the First-Order Loss Function

Jul 16, 2026

Upload by Ivy

The Gaussian Error Linear Unit is usually motivated probabilistically as the average output of an input-dependent Bernoulli gate. This work provides an alternative structural view: GELU equals the expected output of a hard linear gate with a Gaussian random threshold, yielding a generative interpretation for Bernoulli gating. Using a decomposition from stochastic inventory theory, the approach produces a threshold-transmission family including ReLU, GELU, SiLU/Swish, and hard swish, and derives finite-transition piecewise-polynomial variants. Controlled experiments on vision and language models show uniform-threshold gates competitive with standard activations, often improving performance and exploiting the finite transition region.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages18

Reading Time45 min

Introduction

Gaussian first-order loss functions

Signal transmission versus loss accounting

Uncertain threshold interpretation

Threshold-transmission alternatives and relations to gated activations

Computational study

Conclusion

arXiv :2607 .03664v2 [ cs .LG] 10 Jul 2026 A Structural Interpretation of GELU and Threshold-Transmission Activations via the First-Order Loss Function Roberto Rossi Business School, University of Edinburgh, Edinburgh, UK [roberto.rossi@ed.ac.uk](roberto.rossi@ed.ac.uk) Abstract The Gaussian Error Linear Unit is usually motivated as the expected output of an input-dependent Bernoulli gate. This work gives an alternative interpretation: GELU is the expected output of a hard linear gate with a Gaussian random threshold. This view provides a generative interpretation for the Bernoulli gate: the gate opens once the input clears a latent Gaussian threshold. This interpretation stems from a decomposition based on well-known results in stochastic inventory theory and leads to a threshold-transmission family that includes ReLU, GELU, SiLU/Swish, and hard swish as special cases. By considering a latent uniform threshold, we recover a hard-swish-like piecewise-polynomial gate whose nonlinear transition is confined to a finite interval, yielding fixed-and learned-width variants. Controlled experiments on compact vision and language models show that calibrated or learned uniformthreshold gates are consistently competitive with GELU, ReLU, and SiLU/Swish, improve over them in most tested settings, and use the finite transition region nontrivially. 1 Introduction The Gaussian Error Linear Unit [8] is a widely adopted activation function [4, 12] defined as GELU(z) = zΦ(z), where Φ denotes the cumulative distribution function of a standard normal random variable. In [8], the authors define GELU as the expectation of a modification to Adaptive Dropout [1], thus relating it to stochastic regularisers [19] . In particular, they motivate GELU through the following probabilistic view of a neuron’s output. Let B (z) be a Bernoulli random variable with success probability Φ(z), i.e. B (z) ∼ Bernoulli(Φ(z)) . If the input z is retained with probability Φ(z) and dropped with probability 1 − Φ(z), then on average the output is E [zB(z)] = zE[B(z)] = zΦ(z) . Thus GELU can be interpreted as the expected output of an input-dependent Bernoulli gate. This explanation is formally correct and gives a useful regularisation-based motivation. It also supplies a statistical reason for the Gaussian CDF, since neuron inputs are often approximately normal, especially in the presence of Batch Normalisation. However, it leaves open a different structural question: beyond being the average output of a stochastic gate, does the expression zΦ(z) arise from a more general principle? We address this question by making the following contributions: • We identify GELU as the signal-transmission component zΦ(z) of the Gaussian complementary first-order loss function [7, 24], which is typically used in inventory control and finance to measure an expected surplus. • We explain why the remaining truncated-moment correction ϕ (z) is not retained in the activation function: removing it preserves the origin and suppresses confidently negative inputs, while retaining it gives an expected-surplus accounting quantity. • We provide a generative interpretation for the Bernoulli gate: Φ(z) is the probability that the input clears a Gaussian latent threshold; by considering alternative threshold distributions, we derive a threshold-transmission family aF (z) = zF(z) that includes ReLU, SiLU/Swish, and hard swish. • In particular, we recover hard swish [9] by studying uniform-threshold gates, and introduce calibrated and learned-width variants (UELU and TUELU) . • Finally, we provide controlled empirical evidence on compact vision and language models, comparing these threshold-derived gates with standard baselines while reporting closed/transition/open region occupancy. The remainder of this work is structured as follows. Section 2 reviews first-order loss functions and identifies the Gaussian decomposition underlying GELU. Section 3 distinguishes sig

FAQs

How does this paper reinterpret GELU beyond the standard Bernoulli-gate expectation view?

GELU is interpreted as the expected output of a hard linear gate whose threshold is Gaussian random. In this view, the gate opens when the input exceeds a latent Gaussian threshold.

What is the role of the first-order loss function in deriving the GELU structure?

The paper identifies GELU as the signal-transmission component zΦ(z) embedded within the Gaussian complementary first-order loss function. A truncated-moment correction term is discussed as being omitted to preserve origin and suppress confidently negative inputs.

How are threshold-transmission activations related to known activation functions and how are new variants obtained?

By choosing alternative threshold distributions, the framework yields a threshold-transmission family aF(z)=zF(z) that includes ReLU, SiLU/Swish, and hard swish. Studying uniform thresholds recovers hard-swish-like piecewise-polynomial gates and enables calibrated or learned-width variants.

Jul 16, 2026

Upload by Ivy