Improving LLM-Generated Process Model Quality Through Reinforcement Learning: The Role of Reward Function Design

Jul 16, 2026

Upload by Miles

Large language models generate BPMN process models from natural-language descriptions, but supervised fine-tuning restricts outputs to training-set patterns. Reinforcement learning can surpass this limit with external quality measures, yet reward design for multi-dimensional quality has remained unclear. This study systematically examines reward composition for RL-based process model generation using two LLM families (Llama 3.1 8B, Qwen 2.5 14B) across 48 configurations and 38 automated evaluation metrics spanning syntactic, pragmatic, and semantic quality.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages21

Reading Time53 min

Introduction

Reward Function Design for RL-Based Process Modeling

Evaluation Framework and Multi-Dimensional Quality

Improving LLM-Generated Process Model Quality Through Reinforcement Learning: The Role of Reward Function Design Alexander Rombacha,b,∗ , Chantale Lauera,b and Nijat Mehdiyeva,b a German Research Center for Artificial Intelligence (DFKI), Campus D3 2, Saarbrücken, 66123, Saarland, Germany b Saarland University, Campus D3 2, Saarbrücken, 66123, Saarland, Germany arXiv :2607 .06 175v 1 [ cs .CL] 7 Jul 2026 ARTICLE INFO Keywords: Business Process Modeling Large Language Models Supervised Finetuning Reinforcement Learning AB STRACT Large language models (LLMs) can generate BPMN process models from natural-language descriptions, yet supervised fine-tuning (SFT) limits their output quality to the patterns present in the training data. Reinforcement learning (RL) can optimize beyond this ceiling using external quality measures, but how the reward function should be designed when quality is multi-dimensional remains unexplored. We present a systematic investigation of reward function design for RL-based process model generation, training two LLM families (Llama 3.1 8B, Qwen 2.5 14B) under 48 configurations using Group Sequence Policy Optimization with rewards derived from an automated evaluation framework comprising 38 metrics across syntactic, pragmatic, and semantic quality. Three findings emerge. First, RL significantly improves pragmatic and syntactic quality while preserving semantic fidelity, reducing output variability by more than sixfold. Second, equal reward weighting consistently outperforms targeted weighting: emphasizing a specific dimension fails to improve it and can collapse the model into a low-quality mode. Third, design choices interact with model architecture in non-trivial ways: the invalidity penalty is essential for one model but irrelevant for the other, and SFT initialization is indispensable for one architecture but counterproductive for another. These results demonstrate that reward composition is a primary determinant of optimization outcomes, with effects as large as the decision to apply RL itself. The findings generalize to any structured generation task where quality is assessed along multiple automated dimensions. We release our implementation and experimental code at [https://github.com/chlauer99/RL_for_process_modeling](https://github.com/chlauer99/RL_for_process_modeling). 1. Introduction The creation of Business Process Model and Notation (BPMN) models is a complex task requiring both domain knowledge and proficiency in modeling conventions. As organizations increasingly seek to document, analyze, and automate their workflows, the demand for process models has outpaced the availability of trained modelers, creating a bottleneck that limits the scalability of business process management initiatives [19] . Large language models (LLMs) offer a promising path to addressing this bottleneck by enabling the generation of process models directly from natural-language descriptions, thereby lowering the expertise barrier and accelerating the modeling lifecycle [20] . Recent work has demonstrated that LLMs can indeed produce BPMN-like structures from text, with interactive systems supporting iterative refinement [22, 26] and compact intermediate representations improving generation robustness [5, 21] . Systematic benchmarking has further shown that open-source LLMs achieve competitive syntactic and pragmatic quality on BPMN generation tasks, though semantic fidelity and output validity remain significant challenges [27] . Supervised fine-tuning (SFT) with parameterefficient methods such as LoRA [17] has been shown to improve structural correctness and validity, enabling smaller models to outperform larger untuned baselines. However, ∗Corresponding author alexander_michael. [rombach@uni-saarland.de](rombach@uni-saarland.de) (A. Rombach); [chantale.lauer@dfki.de](chantale.lauer@dfki.de) (C. Lauer); [nijat.mehdiyev@dfki.de](nijat.mehdiyev@dfki.de) (N. Mehdiyev) ORCID(s): 0000-0002-91

FAQs

Why does supervised fine-tuning (SFT) limit LLM-generated BPMN quality?

SFT imitates patterns in the training data, so it cannot push the model beyond the demonstrations toward consistently better outputs under new criteria.

How does reinforcement learning (RL) improve BPMN process model generation here?

RL optimizes the model directly against external automated quality measures derived from an evaluation framework, enabling improvements over SFT baselines.

What does the study find about reward function weighting and model behavior?

Equal reward weighting outperforms targeted weighting; emphasizing a specific dimension may fail to improve it and can drive the model into a low-quality mode, with reward components interacting with model architecture in non-trivial ways.

Jul 16, 2026

Upload by Miles