Reinforcement learning with model-based feedforward inputs for robotic table tennis

Aug 25, 2026

The work reframes reinforcement learning by optimizing feedforward inputs rather than feedback policies, aiming to stabilize training and shift most learning effort toward a supervised learning formulation. Labels are generated using a variant of iterative learning control that incorporates prior knowledge about the robot’s dynamics. The framework is evaluated on real-world ping-pong interception and returning with a four-degree-of-freedom pneumatic arm, where control and learning are challenging. Compared with feedback-policy optimization, it improves success rate (100% vs. 96% over 107 trials) and reduces required training samples by about a factor of ten while handling varying incoming trajectories.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages17

Reading Time43 min

AccessFree to download

Abstract

Introduction

Bottlenecks in robotic reinforcement learning

Key motivation for the proposed feedforward framework

Reinforcement learning with model-based feedforward inputs for robotic table tennis Hao Ma1 · Dieter Büchler2 · Bernhard Schölkopf2 · Michael Muehlebach1 Received: 31 January 2023 / Accepted: 18 September 2023 / Published online: 17 October 2023 © The Author(s) 2023 Abstract We rethink the traditional reinforcement learning approach, which is based on optimizing over feedback policies, and propose a new framework that optimizes over feedforward inputs instead. This not only mitigates the risk of destabilizing the system during training but also reduces the bulk of the learning to a supervised learning task. As a result, efﬁcient and well-understood supervised learning techniques can be applied and are tuned using a validation data set. The labels are generated with a variant ofiterative learning control, which also includes prior knowledge about the underlying dynamics. Our framework is applied for intercepting and returning ping-pong balls that are played to a four-degrees-of-freedom robotic arm in real-world experiments. The robot arm is driven by pneumatic artiﬁcial muscles, which makes the control and learning tasks challenging. We highlight the potential of our framework by comparing it to a reinforcement learning approach that optimizes over feedback policies. We ﬁnd that our framework achieves a higher success rate for the returns (100% vs. 96%, on 107 consecutive trials, see [https://youtu.be/kR9jowEH7PY](https://youtu.be/kR9jowEH7PY)) while requiring only about one tenth of the samples during training. We also ﬁnd that our approach is able to deal with a variant of different incoming trajectories. Keywords Reinforcement learning · Iterative learning control · Supervised learning · Table tennis robot · Pneumatic artiﬁcial muscle · Soft robotics 1 Introduction Reinforcement learning has been proven to be highly effective in a variety of contexts. An important example is AlphaGo Zero (Silver et al., 2016, 2017), which managed to completely overpower all human players in the game of Go. Other examples include the work of Oh et al.(2016), Tessleretal.(2017), Firoiuetal.(2017), Kanskyetal.(2017) that focused on video games, the work of Yogatama B Hao Ma[hao.ma@tuebingen.mpg.de](hao.ma@tuebingen.mpg.de) Dieter Büchler [dieter.buechler@tuebingen.mpg.de](dieter.buechler@tuebingen.mpg.de) Bernhard Schölkopf [bernhard.schoelkopf@tuebingen.mpg.de](bernhard.schoelkopf@tuebingen.mpg.de) Michael Muehlebach [michael.muehlebach@tuebingen.mpg.de](michael.muehlebach@tuebingen.mpg.de) 1 Learning and Dynamical Systems, Max Planck Institute for Intelligent Systems, Max-Planck-Ring 4, Tübingen 72076, Germany 2 Empirical Inference, Max Planck Institute for Intelligent Systems, Max-Planck-Ring 4, Tübingen 72076, Germany et al. (2016), Paulus et al. (2017), Zhang and Lapata (2017) that focused on natural language processing, and the work of Liu et al. (2017), Devrim Kaba et al. (2017), Cao et al. (2017), Brunner et al. (2018) that focused on computer vision. Despite these successes, where reinforcement learning agents are shown to compete and outperform humans, researchers have struggled to achieve a similar level of success in robotics applications. We identify the following key bottlenecks, which we believe hinder the application of reinforcement learning to robotic systems. This also motivatesour work, which proposes a new reinforcement learning scheme that addresses some of these shortcomings. The ﬁrst factor is that the lack of prior knowledge causes reinforcement learning algorithms to sometimes apply relatively aggressive feedback policies 1 during training. This has the potential to cause irreversible damage to robotic systems, which are often expensive and require careful maintenance (Moldovan & Abbeel, 2012; Schneider, 1996) . Moreover, these aggressive policies are typically not effec- 1 The term“aggressive feedback policy"refers toa policy that generates actions with the potential to induce har

FAQs

What key change does the framework make compared with traditional reinforcement learning?

It optimizes over feedforward inputs instead of optimizing over feedback policies, which reduces the risk of destabilizing the system during training.

How are the training labels generated in this approach?

Labels are generated with a variant of iterative learning control that also incorporates prior knowledge about the underlying robot dynamics.

How does the method perform in real-world robotic table tennis experiments?

The framework achieves a higher return success rate than a feedback-policy reinforcement learning approach (100% vs. 96% across 107 consecutive trials) and needs about one tenth of the training samples.

Aug 25, 2026

Upload by วิน