A Free From Local Minima Algorithm for Training Regressive MLP Neural Networks - Paper

Aug 5, 2026

Upload by Aria

This article presents an innovative training method for regressive MLP neural networks that is designed to avoid local minima in the loss function. It addresses a key limitation of standard Error-Back-Propagation, where gradient-driven optimization lacks a mechanism to determine whether a reached minimum is local or global. The proposed approach bases training on properties of the training-set distribution and its representation inside the neural network, and evaluates performance on a recognized benchmark.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages10

Reading Time25 min

AccessFree to download

INTRODUCTION

Motivation: local minima and black-box training

Reinterpreting the hidden and output layers

Training algorithm and benchmark results

A Free From Local Minima Algorithm for Training Regressive MLP Neural Networks Augusto Montisci [augusto.montisci@unica.it](augusto.montisci@unica.it) University of Cagliari 09123 via Marengo 2, Cagliari, Italy Corresponding Author: Augusto Montisci Copyright © 2024 Augusto Montisci This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. Abstract In this article an innovative method for training regressive MLP networks is presented, which is not subject to local minima. The Error-Back-Propagation algorithm, proposed by William Hinton-Rummelhart, has had the merit of favoring the development of machine learning techniques, which has permeated every branch of research and technology since the mid- 1980s. This extraordinary success is largely due to the black-box approach, but this same factor was also seen as a limitation, as soon more challenging problems were approached. One of the most critical aspects of the training algorithms was that of local minima of the loss function, typically the mean squared error of the output on the training set. In fact, as the most popular training algorithms are driven by the derivatives of the loss function, there is no possibility to evaluate if a reached minimum is local or global. The algorithm presented in this paper avoids the problem of local minima, as the training is based on the properties of the distribution of the training set, or better on its image internal to the neural network. The performance of the algorithm is shown for a well-known benchmark. 1. INTRODUCTION Even if Machine Learning (ML) includes a multiplicity of paradigms, much different among them, most part can be considered as an evolution of Error Backpropagation algorithm (EBP) of MultiLayer Perceptron (MLP) [1] . The credit of this algorithm consists in the fact that for the first time it was possible to train networks with an intermediate layer, and therefore reproduce non-linear inputoutput relationships. This was as true for classification problems as it was for regression problems. Concerning the latter ones, it has been demonstrated [2], that a Perceptron with a single hidden layer is a universal approximator, however leaving open the problem of determining both the number of neurons needed to solve a specific problem, and how to determine the connection weights. The EBP algorithm offered a tool to determine the value of the parameters, while the determination of the optimal number of neurons is still an open problem. Successively, methods have been presented to address both questions [3–8], but the EBP paradigm, with an important series of variations, still represents the standard of machine learning. This paradigm consists in finding the minimum of a loss function, which is typically given by the output mean squared error with respect to the target 2103 Citation: Augusto Montisci. A Free From Local Minima Algorithm for Training Regressive MLP Neural Networks. Advances in Artificial Intelligence and Machine Learning. 2024;4(1):120 . [https://www.oajaiml.com/ | March 2024](https://www.oajaiml.com/ | March 2024) Augusto Montisci value. All the minimization techniques also developed in contexts other than that of ML have been proposed to solve this problem, but the standard is represented by the use of first and second order descent methods [9], in whose category the EBP itself falls. First order algorithms, such as EBP, have made a comeback with the advent of Deep Learning [10], as the huge number of parameters makes second order methods impractical, even in cases in which approximate expressions of the Hessian are used. The methods based on the derivatives ofthe loss function have the advantage of being simple to implement but lack a criterion that allows to establish whether a stationarity point of the loss function r

FAQs

Why is local minima a critical problem in training regressive MLP networks?

Because the most common objective is the mean squared error, and derivative-based optimization cannot distinguish whether a stationary point is local or global. This makes standard training vulnerable to poor minima.

How does the proposed algorithm avoid local minima?

It trains using the properties of the training set distribution and its image within the neural network, rather than relying solely on the loss-function derivatives. This leverages an internal “feature space” interpretation to guide learning.

What is the paper’s interpretation of the hidden layer and output layer roles?

The hidden layer creates a feature space where the input-output relationship becomes linear, while the output layer enforces constraints via coplanarity across different outputs. The last-layer weights follow from linear regression over the feature-space points.

Aug 5, 2026

Upload by Aria