[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-86430-en":3,"doc-seo-86430-105":30,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":21,"is_downloadable":21,"audit_status":21,"page_count":22,"language":23,"language_code":24,"site_id":25,"html_lang":24,"table_of_contents":26,"faqs":27,"seo_title":13,"seo_description":14,"update_tm":28,"read_time":29},86430,549758252649,"Ivy","https://ap-avatar.wpscdn.com/avatar/8000253669c5317157?_k=1778319167496531819",8,"Research & Report","A Structural Interpretation of GELU and Threshold-Transmission Activations via the First-Order Loss Function","The Gaussian Error Linear Unit is usually motivated probabilistically as the average output of an input-dependent Bernoulli gate. This work provides an alternative structural view: GELU equals the expected output of a hard linear gate with a Gaussian random threshold, yielding a generative interpretation for Bernoulli gating. Using a decomposition from stochastic inventory theory, the approach produces a threshold-transmission family including ReLU, GELU, SiLU/Swish, and hard swish, and derives finite-transition piecewise-polynomial variants. Controlled experiments on vision and language models show uniform-threshold gates competitive with standard activations, often improving performance and exploiting the finite transition region.","arXiv :2607 .03664v2 [ cs .LG] 10 Jul 2026  \nA Structural Interpretation of GELU and Threshold-Transmission Activations via the First-Order Loss Function  \nRoberto Rossi  \nBusiness School, University of Edinburgh, Edinburgh, UK  \n[roberto.rossi@ed.ac.uk](roberto.rossi@ed.ac.uk)  \nAbstract  \nThe Gaussian Error Linear Unit is usually motivated as the expected output of an input-dependent Bernoulli gate. This work gives an alternative interpretation: GELU is the expected output of a hard linear gate with a Gaussian random threshold. This view provides a generative interpretation for the Bernoulli gate: the gate opens once the input clears a latent Gaussian threshold. This interpretation stems from a decomposition based on well-known results in stochastic inventory theory and leads to a threshold-transmission family that includes ReLU, GELU, SiLU/Swish, and hard swish as special cases. By considering a latent uniform threshold, we recover a hard-swish-like piecewise-polynomial gate whose nonlinear transition is confined to a finite interval, yielding fixed-and learned-width variants. Controlled experiments on compact vision and language models show that calibrated or learned uniformthreshold gates are consistently competitive with GELU, ReLU, and SiLU/Swish, improve over them in most tested settings, and use the finite transition region nontrivially.  \n1 Introduction  \nThe Gaussian Error Linear Unit [8] is a widely adopted activation function [4, 12] defined as  \nGELU(z) = zΦ(z),  \nwhere Φ denotes the cumulative distribution function of a standard normal random variable. In [8], the authors define GELU as the expectation of a modification to Adaptive Dropout [1], thus relating it to stochastic regularisers [19] . In particular, they motivate GELU through the following probabilistic view of a neuron’s output. Let B (z) be a Bernoulli random variable with success probability Φ(z), i.e. B (z) ∼ Bernoulli(Φ(z)) . If the input z is retained with probability Φ(z) and dropped with probability 1 − Φ(z), then on average the output is  \nE [zB(z)] = zE[B(z)] = zΦ(z) .  \nThus GELU can be interpreted as the expected output of an input-dependent Bernoulli gate. This explanation is formally correct and gives a useful regularisation-based motivation. It also  \nsupplies a statistical reason for the Gaussian CDF, since neuron inputs are often approximately normal, especially in the presence of Batch Normalisation. However, it leaves open a different structural question: beyond being the average output of a stochastic gate, does the expression zΦ(z) arise from a more general principle?  \nWe address this question by making the following contributions:  \n• We identify GELU as the signal-transmission component zΦ(z) of the Gaussian complementary first-order loss function [7, 24], which is typically used in inventory control and finance to measure an expected surplus.  \n• We explain why the remaining truncated-moment correction ϕ (z) is not retained in the activation function: removing it preserves the origin and suppresses confidently negative inputs, while retaining it gives an expected-surplus accounting quantity.  \n• We provide a generative interpretation for the Bernoulli gate: Φ(z) is the probability that the input clears a Gaussian latent threshold; by considering alternative threshold distributions, we derive a threshold-transmission family aF (z) = zF(z) that includes ReLU, SiLU/Swish, and hard swish.  \n• In particular, we recover hard swish [9] by studying uniform-threshold gates, and introduce calibrated and learned-width variants (UELU and TUELU) .  \n• Finally, we provide controlled empirical evidence on compact vision and language models, comparing these threshold-derived gates with standard baselines while reporting closed/transition/open region occupancy.  \nThe remainder of this work is structured as follows. Section 2 reviews first-order loss functions and identifies the Gaussian decomposition underlying GELU. Section 3 distinguishes sig","cbCainP53Vo8fP1o","https://ap.wps.com/l/cbCainP53Vo8fP1o","pdf",3710540,4,1,18,"English","en",105,"# Introduction\n## Gaussian first-order loss functions\n## Signal transmission versus loss accounting\n# Uncertain threshold interpretation\n## Threshold-transmission alternatives and relations to gated activations\n# Computational study\n# Conclusion","[{\"question\":\"How does this paper reinterpret GELU beyond the standard Bernoulli-gate expectation view?\",\"answer\":\"GELU is interpreted as the expected output of a hard linear gate whose threshold is Gaussian random. In this view, the gate opens when the input exceeds a latent Gaussian threshold.\"},{\"question\":\"What is the role of the first-order loss function in deriving the GELU structure?\",\"answer\":\"The paper identifies GELU as the signal-transmission component zΦ(z) embedded within the Gaussian complementary first-order loss function. A truncated-moment correction term is discussed as being omitted to preserve origin and suppress confidently negative inputs.\"},{\"question\":\"How are threshold-transmission activations related to known activation functions and how are new variants obtained?\",\"answer\":\"By choosing alternative threshold distributions, the framework yields a threshold-transmission family aF(z)=zF(z) that includes ReLU, SiLU/Swish, and hard swish. Studying uniform thresholds recovers hard-swish-like piecewise-polynomial gates and enables calibrated or learned-width variants.\"}]",1784211697,45,{"code":4,"msg":31,"data":32},"ok",{"site_id":25,"language":24,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":28},"a-structural-interpretation-of-gelu-and-threshold-transmission-activations-via-the-first-order-loss-function","",{"@graph":36,"@context":85},[37,53,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,51],{"item":41,"name":42,"@type":43,"position":21},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":50},"https://docshare.wps.com/document/research-report/",3,{"item":52,"name":13,"@type":43,"position":20},"https://docshare.wps.com/document/a-structural-interpretation-of-gelu-and-threshold-transmission-activations-via-the-first-order-loss-function/86430/",{"url":52,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":24,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":41,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-27","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"How does this paper reinterpret GELU beyond the standard Bernoulli-gate expectation view?","Question",{"text":75,"@type":76},"GELU is interpreted as the expected output of a hard linear gate whose threshold is Gaussian random. In this view, the gate opens when the input exceeds a latent Gaussian threshold.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"What is the role of the first-order loss function in deriving the GELU structure?",{"text":80,"@type":76},"The paper identifies GELU as the signal-transmission component zΦ(z) embedded within the Gaussian complementary first-order loss function. A truncated-moment correction term is discussed as being omitted to preserve origin and suppress confidently negative inputs.",{"name":82,"@type":73,"acceptedAnswer":83},"How are threshold-transmission activations related to known activation functions and how are new variants obtained?",{"text":84,"@type":76},"By choosing alternative threshold distributions, the framework yields a threshold-transmission family aF(z)=zF(z) that includes ReLU, SiLU/Swish, and hard swish. Studying uniform thresholds recovers hard-swish-like piecewise-polynomial gates and enables calibrated or learned-width variants.","https://schema.org",{"og:url":52,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":52},"index,follow",{"doc_id":7,"site_id":25},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":21,"doc_module":4,"doc_module_name":46,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":20,"doc_module":4,"doc_module_name":46,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":46,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":46,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":46,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":46,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":46,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":46,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":46,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]