A Stitch in Time - Autonomous Model Management via Reinforcement Learning - Concept Drift Model Retraining

Sep 5, 2026

Upload by Lute

Concept drift poses a persistent threat to maintaining high-performing learned models in autonomous systems when data dynamics change over time. Instead of relying on outdated models, discarding history, or retraining with limited new data, the paper frames model management as a sequential decision-making problem. It formalizes the response as a Markov Decision Process and uses reinforcement learning to learn a robust policy for updating models. Results on one synthetic and two real-world datasets show promising performance and a trade-off between learning costs and long-term performance.

LanguageEnglish

CategoryPresentations

FormatPDF

Pages9

Reading Time3 min

AccessFree to download

Introduction

Concept drift and model retraining

Contributions and approach

Related Work

Concept drift methods

Online learning and lifelong learning

Transfer learning connections

A Stitch in Time-Autonomous Model Management via Reinforcement Learning Elad Liebman The University of Texas at Austin [eladlieb@cs.utexas.edu](eladlieb@cs.utexas.edu) Eric Zavesky AT&T Research [ezavesky@research.att.com](ezavesky@research.att.com) Peter Stone The University of Texas at Austin [pstone@cs.utexas.edu](pstone@cs.utexas.edu) ABSTRACT Concept drift-a change, either sudden or gradual, in the underlying properties of data-is one of the most prevalent challenges to maintaining high-performing learned models over time in autonomous systems. In the face of concept drift, one can hope that the old model is sufficiently representative of the new data despite the concept drift, one can discard the old data and retrain a new model with (often limited) new data, or one can use transfer learning methods to combine the old data with the new to create an updated model. Which of these three options is chosen affects not only near-term decisions, but also future needs to transfer or retrain. In this paper, we thus model response to concept drift as a sequential decision making problem and formally frame it as a Markov Decision Process. Our reinforcement learning approach to the problem shows promising results on one synthetic and two real-world datasets. KEYWORDS Reinforcement Learning; Model Retraining; Concept Drift ACM Reference Format: Elad Liebman, Eric Zavesky, and Peter Stone. 2018. A Stitch in Time Autonomous Model Management via Reinforcement Learning. In Proc. of the 17th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2018), Stockholm, Sweden, July 10–15, 2018, IFAAMAS, 9 pages. 1 INTRODUCTION As automation grows, more and more industry control systems around us make decisions autonomously, ranging from image understanding [19] to movie recommendation systems [9] and network and service virtualization [3]. Underneath their hoods, many of these systems rely on models. These models can be descriptive (capturing the properties of data) or predictive (using known data to predict other, latent properties) . However, such systems are often susceptible to the nonstationary, ever-changing dynamics of data in the real world. In recommendation systems, tastes and fashion change. In climate prediction, the properties of the environment change constantly. These changes, either gradual or abrupt, are commonly referred to as concept drift. Such shifts in the feature distribution and underlying label correspondence constitute a significant challenge to learning systems. In the face of concept drift, we consider the problem of how to generically and adaptively adjust models to mitigate the risks of drift. We refer to this challenge as model retraining, or model management. In this paper we propose a novel reinforcement learning (RL) approach, framing the model Proc. of the 17th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2018), M. Dastani, G. Sukthankar, E. André, S. Koenig (eds.), July 10–15, 2018, Stockholm, Sweden. © 2018 International Foundation for Autonomous Agents and Multiagent Systems ([www.ifaamas.org](www.ifaamas.org)). All rights reserved. retraining problem as a sequential decision making task, and harnessing ideas from the RL literature to learn a robust policy for model update. This paper makes two main contributions. First, we formally frame the autonomous model retraining problem as a reinforcement learning task. To our knowledge, we are the first to consider the benefits of this approach. Second, we propose a robust and general framework for learning an autonomous model adaptation policy that balances learning costs with overall performance and is completely decoupled from the underlying models. We show empirically that this approach is superior to other baseline metalearning policies, including two competitive drift detection methods (DDM[7] and HDDM[6]) . We also show how our approach can be scaled to more complex, mult

FAQs

What challenge does the paper address in autonomous systems?

The paper addresses concept drift, meaning sudden or gradual changes in the properties of data that can degrade learned model performance over time.

How does the paper model autonomous model management?

It formulates the model retraining response to concept drift as a sequential decision-making problem and formally frames it as a Markov Decision Process.

What are the reported results and datasets?

The reinforcement learning approach shows promising results on one synthetic dataset and two real-world datasets, demonstrating effectiveness beyond baseline policies.

Sep 5, 2026

Upload by Lute