Data Mining WorkFlow Templates - Intelligent Discovery Assistance and Auto-Experimentation

Sep 12, 2026

Knowledge Discovery in Databases (KDD) has expanded rapidly, yet assisting users in constructing workflows remains difficult. Current KDD systems offer many operators, often leading workflows to grow to hundreds of steps with repeated components. Manual construction becomes error-prone, and workflows are not validated for correctness before execution, causing runtime failures after hours. The paper proposes a knowledge-based workflow representation and workflow templates that combine executable operators and refinable tasks to structure discovery and improve reuse.

LanguageEnglish

CategoryPresentations

FormatPDF

Pages12

Reading Time4 min

AccessFree to download

Abstract

Introduction

Challenges in Existing KDD Workflow Construction

Limitations of Planning-Based Approaches

Proposed Solution: Knowledge-Based Representation and Workflow Templates

Data Mining Work􀀍ow Templates for Intelligent Discovery Assistance and Auto-Experimentation J􀁿org-Uwe Kietz 1 , Floarea Serban 1 , Abraham Bernstein 1 , and Simon Fischer2 1 University of Zurich, Department of Informatics, Dynamic and Distributed Information Systems Group, Binzm􀁿uhlestrasse 14, CH-8050 Zurich, Switzerland [f](fkietz|serban|bernsteing@ifi.uzh.ch)[kietz|serban|bernstein](fkietz|serban|bernsteing@ifi.uzh.ch)[g](fkietz|serban|bernsteing@ifi.uzh.ch)[@ifi.uzh.ch](fkietz|serban|bernsteing@ifi.uzh.ch) 2 Rapid-I GmbH, Stockumer Str. 475, 44227 Dortmund, Germany [fischer@rapid-i.com](fischer@rapid-i.com) Abstract. Knowledge Discovery in Databases (KDD) has grown a lot during the last years. But providing user support for constructing work-􀀍ows is still problematic. The large numbers of operators available in current KDD systems makes it di􀀎cult for a user to successfully solve her task. Also, work􀀍ows can easily reach a huge number of operators (hundreds) and parts of the work􀀍ows are applied several times. Therefore, it becomes hard for the user to construct them manually. In addition, work􀀍ows are not checked for correctness before execution. Hence, it frequently happens that the execution of the work􀀍ow stops with an error after several hours runtime. In this paper3 we present a solution to these problems. We introduce a knowledge-based representation ofKDD work􀀍ows as a basis for cooperative-interactive planning. Moreover, we discuss work􀀍ow templates, i.e. abstract work􀀍ows that can mix executable operators and tasks to be re􀀌ned later into sub-work􀀍ows. This new representation helps users to structure and handle work􀀍ows, as it constrains the number of operators that need to be considered. Finally, work􀀍ows can be grouped in templates which foster re-use further simplifying KDD work􀀍ow construction. 1 Introduction One of the challenges of Knowledge Discovery in Database (KDD) is assisting the users in creating and executing KDD work􀀍ows. Existing KDD systems such as the commercial Clementine and Enterprise Miner or the open-source Weka, MiningMart, KNIME and RapidMiner support the user with nice graphical user interfaces, where operators can be dropped as nodes onto the working pane and 3 This paper reports on work in progress. Refer to [http://www.e-lico.eu/eProPlan to](http://www.e-lico.eu/eProPlan to)[ ](http://www.e-lico.eu/eProPlan to)see the current state of the Data Mining ontology for WorkFlow planning (DMWF), the IDA-API, and the eProPlan Protege plug-ins we built to model the DMWF. The RapidMiner IDA-wizard will be part of a future release of RapidMiner check [http://www.rapidminer.com/ for](http://www.rapidminer.com/ for) it. the data-􀀍ow be speci􀀌ed by connecting the operator-nodes. This works very well as long as neither the work􀀍ow becomes too complicated and large nor the number of operators becomes too large. The number of operators in such systems, however, has been growing fast. All of them contain over 100 operators and RapidMiner, which includes Weka, even over 600 . It can be expected that with the incorporation of text-, image-, and multimedia-mining as well as the transition from closed systems with a 􀀌xed set of operators to open systems that can also use Web services as operator (which is especially interesting for domain speci􀀌c data access and transformations) will further accelerate the rate of growth resulting in total confusion for most users. Not only the number of operators, but also the size of the work􀀍ows is growing. Today's work􀀍ows can easily contain hundreds of operators. Parts of the work􀀍ows are applied several times ( e.g. the preprocessing sub-work􀀍ow has to be applied on training, testing, and application data) implying that the users either need to copy/paste or even design a new sub-work􀀍ow4 several times. None of the systems maintain this \copy"-relationship, it is left to the user to maintain the relationship in the light of changes. Anothe

FAQs

Why are KDD workflows hard to construct in existing systems?

They include very large numbers of operators and can grow to hundreds of steps, often reusing sub-workflows repeatedly. This makes manual building and management difficult for users.

What problem occurs when workflows are executed without prior correctness checks?

Small syntactic incompatibilities between operators and the input data can stop execution with errors after several hours of runtime.

How do workflow templates and the proposed representation help?

Workflow templates act as abstract workflows combining executable operators and tasks to be refined later. The knowledge-based representation constrains operator choices, improving workflow structuring and reuse.

Sep 12, 2026

Upload by Alex Sinclair