Is Model Instability Just Noise to be Tolerated or a Property that can be Managed

Jul 16, 2026

Software analytics rerunning the same analysis can produce different models and conclusions, reducing trust and limiting adoption in practice. A large study across 127 multi-objective software engineering optimization problems (12,700 test cases) shows repeated runs of a state-of-the-art optimizer agree on only 13.7% of test cases. The work argues instability is measurable and manageable, not merely noise: refining label spending, model complexity, and split scoring increases agreement 4.8× and reduces average optimization error standard deviation by 22% while improving recommendation quality. Causal and data-locality interventions help only partially, indicating data-imposed stability limits. Instability should be reported as a standard evaluation axis.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages12

Reading Time30 min

Introduction

Research questions and contributions

Is Model Instability just Noise to be Tolerated or a Property that can be Managed? Amirali Rayegan North Carolina State University Raleigh, North Carolina, USA [arayega@ncsu.edu](arayega@ncsu.edu) Lunxiao Li North Carolina State University Raleigh, North Carolina, USA [lli66@ncsu.edu](lli66@ncsu.edu) Tim Menzies North Carolina State University Raleigh, North Carolina, USA [tjmenzie@ncsu.edu](tjmenzie@ncsu.edu) arXiv :2607 . 10420v 1 [ cs . SE] 11 Jul 2026 Abstract—In software analytics, rerunning the same analysis twice often yields different models and conclusions. This reduces trust in the model and limits its use. We find that model instability is a major problem. Across 127 multi-objective SE optimization problems (12,700 test cases), repeated runs of a state-of-theart optimizer agree on only 13.7% of test cases, even under improved settings. We argue that this instability is not merely noise to tolerate, but a property that can be measured and managed. By adjusting how labels are spent, how complex the models become, and how splits are scored, we obtain models that agree 4.8 times as often as the default configuration. The standard deviation of optimization error falls by 22% on average (mean std 17.4 to 13.6), while recommendation quality improves rather than degrades. In terms of quality, the refined settings are statistically top-ranked on 119 of 127 datasets, compared to 74 for the defaults. We then test causal and data-locality interventionsand find that they help only partially, suggesting a residual stability floor. Our evidence suggests there are fundamental limits to stability set by the data itself (noise, scarce labels, proxy objectives, and the many near-equivalent models a dataset admits). We conclude that instability should be treated as a standard evaluation axis in SE optimization, which should be routinely measured, reported alongside performance, and used to calibrate trust in any single run. The methods in this paper provide a baseline against which future efforts to reduce SBSE instability can be judged. To support open science, we offer the following reproduction package: [https://tinyurl.com/Model-Instability](https://tinyurl.com/Model-Instability) Index Terms—Software Analytics, Rashomon Effect, SearchBased Software Engineering, Optimization, Causal Reasoning I. INTRODUCTION Software analytics is an influential research area with much practical value [1]–[6] . By mining historical project data, teams allocate testing effort, prioritize high-risk modules, and cut maintenance costs. Eventually, though, stakeholders aska deceptively simple question: “What did you learn from all that data?” The answer is usually a symbolic explanation, a decision tree, a rule set, or a causal graph. Unfortunately, these explanations are often unstable. Rerunning the same analysis yields different models and, therefore, different conclusions. This reduces trust in the model and, therefore, limits its use. For example, Figure 2 shows model variability seen after running the same optimizer several times on the same data using different random number seeds. As seen in Figure 2, the same learner can produce models with different structures and use different attributes. The problem of model instability is not just a quirk of the learner used in Figure 2 . Rather, it is endemic. As SE problems grow larger and larger, state-of-the-art methods inject randomness to scale, and conclusion instability has been reported across regression [7], text mining [8], defect prediction [9], causal discovery [10], and LLMs [11] . A natural reaction is to treat instability as a bug to fix with more data, tuning, or ensembling. But the literature suggests a harder truth. Often, there is no single “best” explanation to recover. Breiman’s “two cultures” essay notes that structurally different models can fit the same data nearly as well [13], the Rashomon effect, which Xin et al. formalize as the (often enormous) Rashomo

FAQs

What problem does the paper identify in software analytics and SE optimization?

Repeated runs of the same analysis often produce different models and conclusions. This creates performance and structural instability that reduces trust and limits practical use.

How prevalent is performance instability according to the study?

Across 127 multi-objective SE optimization problems and 12,700 test cases, repeated runs agree on only 13.7% of test cases even under improved settings.

How can instability be managed, and what evidence is provided about its limits?

Adjusting label spending, model complexity, and split scoring yields models that agree 4.8 times more often than the default configuration, and reduces optimization error standard deviation by 22% on average. Causal and data-locality interventions help only partially, suggesting a residual stability floor driven by the data itself.

Jul 16, 2026

Upload by Evangeline