Trajectory Ensembles and Machine Learning - From reinforcement learning for rare event sampling to training of neural network ensembles

Aug 4, 2026

Upload by Maeve

This PhD thesis connects statistical mechanics with machine learning, aiming to bridge trajectory ensemble techniques and modern deep learning methods. It develops two main avenues: translating rare-event sampling into reinforcement learning to exploit deep RL for near-optimal sampling dynamics, and introducing a trajectory-sampling algorithm that jointly trains an ensemble of neural networks. The work derives analysis for finite and infinite time problems. It also proposes a coupled ensemble training method using Markov chain Monte Carlo with gradient-free updates, outperforming uncoupled neuroevolution baselines on linear analysis and empirical MNIST experiments.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages163

Reading Time6 h 51 min

AccessFree to download

Introduction

Supervised Learning

Linear Models

Neural Networks

Training as an Optimisation Problem

Reinforcement Learning

Mathematical Definition

Grid World

Dynamic Programming

Model-Free Methods

Approximate Solution Methods

Policy Gradient Methods

Large Deviation Theory

Monte Carlo Sampling

Monte Carlo Integration

Importance Sampling

Markov-Chain Monte Carlo Methods

Supporting Computational Infrastructure

Course: High Performance Computing in Julia

Package: Experimenter . jl

Package: TransitionPathSampling . jl

Package: SimpleNNs . jl

A Reinforcement Learning Approach to Rare Trajectory Sampling

Training Neural Network Ensembles via Trajectory Sampling

Minibatch Training of Neural Network Ensembles via Trajectory Sampling

Conclusions & Outlook

Trajectory Ensembles and Machine Learning: From reinforcement learning for rare event sampling to training of neural network ensembles by Jamie F. Mair A thesis submitted for fulfilment of the requirements for the degree of Doctor of Philosophy in the School of Physics & Astronomy University of Nottingham Supervisors: Prof. Juan P. Garrahan Dr. Adam Moss June 2023 Abstract This thesis builds on the existing body of work connecting the fields of statistical mechanics and machine learning. Many advances in machine learning have found their roots in statistical mechanics, e.g. simulated annealing for heuristic optimisation. Primarily, we aim to build bridges between trajectory ensemble techniques and current advances in machine learning, and in particular, deep learning. We explore these connections in two avenues: the first connects the study of rare events with that of reinforcement learning (RL); the second introduces a trajectory sampling algorithm for jointly training an ensemble of neural networks. We derive a framework for translating a rare event sampling problem into the language of RL, offering a way to leverage modern deep RL algorithms to obtain near-optimal sampling dynamics. Our work showcases a plethora of these RL algorithms and provides analysis on examples, including both finite and infinite time problems. Furthermore, we present a novel neural network ensemble training method, facilitated via Markov chain Monte Carlo algorithms, to produce coupled ensembles using gradient-free updates. We show that our coupled ensembles perform better, and are easier to train, than their uncoupled counterparts trained via neuroevolution, providing both analytic results on a linear problem and empirical evidence on the MNIST problem. Contents Acronyms i Introduction iii 1 Introduction to Machine Learning 1 1.1 Supervised Learning ............................. 2 1.2 Linear Models ................................. 3 1.3 Neural Networks ............................... 4 1.3.1 Activation Functions ......................... 5 1.3.2 Back-Propagation ........................... 5 1.4 Training as an Optimisation Problem .................... 8 1.4.1 Gradient Descent ........................... 8 1.4.2 Stochastic Gradient Descent ..................... 9 2 Reinforcement Learning 11 2.1 Mathematical Definition ........................... 12 2.2 Grid World .................................. 16 2.3 Dynamic Programming ............................ 19 2.3.1 Bellman Equations and Optimality ................. 19 2.3.2 Policy Evaluation ........................... 20 2.3.3 Policy Iteration ............................ 22 2.4 Model-Free Methods ............................. 24 2.4.1 Policy Evaluation through Interaction ................ 25 2.4.2 Monte Carlo Policy Iteration ..................... 27 2.4.3 On/Off Policy Methods ........................ 28 2.4.4 Bootstrapping: Updating Estimates with Estimates ........ 32 2.4.5 n-step Temporal Difference ..................... 32 2.5 Approximate Solution Methods ....................... 34 2.5.1 Large State Spaces and Generalisation ............... 35 2.5.2 The Deadly Triad ........................... 35 2.6 Policy Gradient Methods ........................... 36 2.6.1 Parameterising a Policy ....................... 36 2.6.2 Policy Gradient: REINFORCE ................... 38 2.6.3 Variance of Vanilla Policy Gradient ................. 40 2.6.4 REINFORCE with Value Baseline .................. 45 2.6.5 Actor-Critic .............................. 46 3 Large Deviation Theory 49 4 CONTENTS 3.1 Gärtner-Ellis Theorem ............................ 50 3.2 Moments of the SCGF ............................ 51 3.2.1 Example: Bernoulli Random Variables ............... 51 3.3 Dynamical Large Deviations ......................... 53 3.3.1 Doob Dynamics ............................ 56 4 Monte Carlo Sampling 57 4.1 Mo

FAQs

How does the thesis link rare event sampling with reinforcement learning?

It derives a framework translating rare-event sampling into the language of reinforcement learning, enabling use of modern deep RL algorithms to obtain near-optimal sampling dynamics.

What is the second main direction of the thesis?

It presents a trajectory sampling algorithm for jointly training an ensemble of neural networks, including methods for minibatch training.

How are coupled neural network ensembles trained, and what evidence supports their advantage?

Coupled ensembles are trained using Markov chain Monte Carlo facilitated, with gradient-free updates; results show improved performance and easier training versus uncoupled ensembles, supported analytically on a linear problem and empirically on MNIST.

Aug 4, 2026

Upload by Maeve