Deep Black-Box Reinforcement Learning with Movement Primitives

Aug 2, 2026

Upload by Logic

Episode-based reinforcement learning treats reinforcement learning as a black-box optimization problem, learning controller parameters such as movement primitives conditioned on a context descriptor. The approach enables smooth control trajectories, supports non-Markovian reward definitions, and performs exploration directly in movement-primitive parameter space, making it attractive for sparse-reward tasks. High-dimensional primitive parameters have limited deep learning usage, motivating a new deep episode-based RL algorithm.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages22

Reading Time55 min

AccessFree to download

Introduction

Episode-based vs step-based reinforcement learning

Movement primitives and context adaptation

Proposed deep ERL method

Differentiable trust region layers and exact policy updates

Reward formulations and comparative evaluation

Experiments and results

Dense, sparse, and non-Markovian reward settings

Deep Black-Box Reinforcement Learning with Movement Primitives Fabian Otto 1 ;2 , Onur Celik3 , Hongyi Zhou3 , Hanna Ziesche 1 , Ngo Anh Vien 1 , and Gerhard Neumann3 1 Bosch Center for Artiﬁcial Intelligence, Germany 2 University of T¨ubingen, Germany 3 Autonomous Learning Robots, Karlsruhe Institute of Technology, Germany [fabian.otto@bosch.com](fabian.otto@bosch.com) Abstract: Episode-based reinforcement learning (ERL) algorithms treat reinforcement learning (RL) as a black-box optimization problem where we learn to select a parameter vector of a controller, often represented as a movement primitive, for a given task descriptor called a context. ERL offers several distinct beneﬁts in comparison to step-based RL. It generates smooth control trajectories, can handle non-Markovian reward deﬁnitions, and the resulting exploration in parameter space is well suited for solving sparse reward settings. Yet, the high dimensionality of the movement primitive parameters has so far hampered the effective use of deep RL methods. In this paper, we present a new algorithm for deep ERL. It is based on differentiable trust region layers, a successful on-policy deep RL algorithm. These layers allow us to specify trust regions for the policy update that are solved exactly for each state using convex optimization, which enables policies learning with the high precision required for the ERL. We compare our ERL algorithm to state-of-the-art step-based algorithms in many complex simulated robotic control tasks. In doing so, we investigate different reward formulations -dense, sparse, and non-Markovian. While step-based algorithms perform well only on dense rewards, ERL performs favorably on sparse and non-Markovian rewards. Moreover, our results show that the sparse and the non-Markovian rewards are also often better suited to deﬁne the desired behavior, allowing us to obtain considerably higher quality policies compared to step-based RL. Keywords: Movement Primitives, Deep Episode-Based RL, Trust Region Layers 1 Introduction Reinforcement learning (RL) problems can be viewed from a step-based [1, 2, 3] and an episodebased perspective [4, 5, 6] . In the former, most commonly found in deep RL, a policy selectsan action for each state of the trajectory. Consequently, step-based RL requires Markovian reward deﬁnitions. Moreover, the exploration in action space often induces a random walk that inadequately explores the entire behavior space of the agent. As a result, most step-based RL algorithms only work well with dense rewards, where the agent receives a reward signal at each time step, rather than only at the last step. We refer to the latter as a sparse reward setting. In the episode-based reinforcement learning (ERL) perspective, we choose the complete behavior during the episode with respect to the controller parameters in the beginning of the episode. Typically, simple controllers are used that are valid only for executing a single trajectory, such as movement primitives (MPs) . At the beginning of an episode, the MP parameters are adapted to a context vector that serves as task descriptor and may contain e. g. different initial joint conﬁgurations, the goal to reach, or start positions as well as object and obstacles locations. ERL allows for efﬁcient exploration of the behavior space because exploration is implemented in the parameter space of the MP, allowing these algorithms to learn from sparse or even non-Markovian rewards. In addition, ERL inherently generates smooth control trajectories due to the use of MPs, which often leads to more energy efﬁcient behavior. However, ERL has so far not gained much popularity as 6th Conference on Robot Learning (CoRL 2022), Auckland, New Zealand. for t in 1; 2; : : : ; T Context c Policy 􀀙􀀒 (w jc) Desired Trajectory 􀀜 d = 􀀉(w) = (sd1 ; sd2 ; : : : ; sdT) Controller f (st ; sdt) at st+1 Environment Figure 1: Overview of the proposed Black-Box Reinforce

FAQs

What is episode-based reinforcement learning (ERL) in this work?

ERL frames reinforcement learning as black-box optimization over controller parameters at the start of each episode, conditioned on a context vector rather than selecting actions at every step.

Why are movement primitives useful for ERL?

Movement primitives parameterize a whole trajectory and yield smooth control behavior; adapting primitive parameters to context enables efficient exploration in parameter space and supports sparse or non-Markovian rewards.

How does the proposed deep ERL method differ from prior deep step-based RL?

It adapts differentiable trust region projection layers to the ERL setting, providing exact trust regions via convex optimization per state, which improves precision for learning in high-dimensional parameter spaces.

Aug 2, 2026

Upload by Logic