Stratified Sampling Algorithms for Machine Learning Methods in Solving Two-scale Partial Differential Equations - Key vanishing gradient - improved numerical solutions

Aug 4, 2026

Upload by Noah

Partial differential equations (PDEs) with multiple scales or defined on large domains are common across science and engineering, but numerical approximation is often difficult due to scale-dependent solution behavior. Machine learning methods increasingly approximate PDEs, yet many approaches still emphasize small domains. This work studies two-scale solutions that are highly localized in some regions and nearly flat in others, requiring effective sampling on a large domain.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages35

Reading Time1 h 28 min

AccessFree to download

Introduction

Multi-scale phenomena and multi-scale PDEs

Challenges in numerical solving with large domains

Motivation from vanishing gradients in neural networks

Problem setup and targeted PDEs

Two-scale structures: localized and flat regions

Boundary effects and need for large domains

Fisher and Zeldovich equations

Proposed stratified sampling approach

Gradient behavior over long-range flat regions

Comparison with uniform random classical sampling

Numerical results and findings

Improved accuracy and solution consistency

Stratified Sampling Algorithms for Machine Learning Methods in Solving Two-scale Partial Differential Equations Eddel Elí Ojeda Avilés a,b , Daniel Olmos Liceaga a,b and Jae-Hun Jung b,∗ a Universidad de Sonora, Blvd. Luis Encinas y Rosales S/N, 83000, Hermosillo, Sonora, México b Pohang University of Science and Technology, Pohang, 37673, Korea | ARTICLE INFO | AB STRACT | | --- | --- | | Keywords:
Stratified sampling
Physics-Informed Neural Networks Reaction-Diffusion equations Vanishing gradient problem Multi-scale partial differential equations | Partial differential equations (PDEs) with multiple scales or those defined over sufficiently large domains arise in various areas of science and engineering and often present problems when approximating the solutions numerically. Machine learning techniques are a relatively recent method for solving PDEs. Despite the increasing number of machine learning strategies developed to approximate PDEs, many remain focused on relatively small domains. When scaling the equations, a large domain is naturally obtained, especially when the solution exhibits multiscale characteristics. This study examines two-scale equations whose solution structures exhibit distinct characteristics: highly localized in some regions and significantly flat in others. These two regions must be adequately addressed over a large domain to approximate the solution more accurately. We focus on the vanishing gradient problem given by the diminishing gradient zone of the activation function over large domains and propose a stratified sampling algorithm to address this problem. We compare the uniform random classical sampling method over the entire domain and the proposed stratified sampling method. The numerical results confirm that the proposed method yields more accurate and consistent solutions than classical methods. | 1. Introduction Multiple spatial and temporal scales characterize numerous phenomena in science and engineering. Some examples of such phenomena are presented in climate and ocean sciences [1], materials sciences [2], thermal dynamics [3], and other fields. The typical mathematical descriptions of such phenomena involve non-linear partial differential equations (PDEs) with multiple scales. This situation has led to the development of the multi-scale method or analysis, comprising various techniques employed to construct approximations of the solutions to problems that depend on different scales simultaneously [4, 5] . However, numerically solving problems involving multiple scales is generally problematic. One approach involves introducing fast- and slow-scaling variables and treating them as independent. While it is challenging, the recent development of strategies for machine learning methods enables the numerical approximation of solutions to such problems by training neural networks [6–8] that directly simulate the given PDEs. This paper, examines two-scale equations whose solution structures have distinct characteristics: smooth and highly localized in some regions and significantly flat in other regions. The considered PDEs in this paper include the Fisher equation, a model for the spatial and temporal spread of an advantageous allele in a one-dimensional medium [9], and the Zeldovich equation, which describes the propagation of a flame in the combustion and detonation of gases [10, 11] . The solutions to these equations are highly localized in small regions and nearly flat in most areas. Due to the localization of the solution, the computational domain could be defined in a small area with numerous sampling points. However, the numerical solution becomes highly sensitive to the boundary values because of the highly nonlinear terms in the equation although the solution changes slowly near the truncated boundaries. Simply imposing exact boundary values at the truncated domain boundaries is insufficient. Thus, defining a sufficiently large computational domain is necessary so

FAQs

Why do two-scale PDE problems require careful sampling on large domains?

Two-scale solutions are localized in some regions but nearly flat elsewhere. A sufficiently large computational domain reduces boundary truncation effects while maintaining accuracy over the flat regions.

What is the vanishing gradient problem in this context?

When the solution is flat over long-range regions, neural network activation functions produce very small derivatives. Gradients then vanish, causing slow or stalled loss decay during training.

How does the proposed stratified sampling method differ from uniform random sampling?

The method uses stratified sampling to address the distinct solution structure across regions. Numerical results show higher accuracy and more consistent solutions than uniform random classical sampling.

Aug 4, 2026

Upload by Noah