A Single Diffusion-Policy Controller for Multi-Task Block Pushing with Zero-Shot Sim-to-Real Transfer

Jul 16, 2026

Upload by Sage

Diffusion policies provide an effective way to learn complex robotic maneuvers from behavior cloning, yet extending them to multi-task manipulation and sim-to-real transfer remains challenging. This work trains a single diffusion policy from scratch using reinforcement learning for multi-task block pushing with varying block shapes. A simplified policy loss reweights the BC-style evidence lower bound and becomes an RL learning module. Exploration without demonstrations is addressed via reverse curriculum generation and objective-centric representations, with evaluations testing zero-shot transfer across goals, shapes, weights, and surface friction.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages8

Reading Time20 min

Introduction

Motivation: from single-task to universal control

Limits of BC-based diffusion training

Online RL for diffusion policies and sim-to-real challenge

Contribution and pipeline overview

A Single Diffusion-Policy Controller for Multi-Task Block Pushing with Zero-Shot Sim-to-Real Transfer Haitong Ma 1 , Haldun Balim 1 , Yang Hu 1 , Bo Dai2 and Na Li 1 arXiv :2607 . 10892v1 [ cs .RO] 12 Jul 2026 Abstract—Diffusion policies have shown promising empirical performance in representing and learning complex maneuvers for robots using behavior cloning (BC). In this paper, we explore training diffusion policies from scratch using reinforcement learning (RL) for multi-task robotic manipulation. Specifically, we aim to train a single diffusion policy for block-pushing tasks with multiple shapes. The proposed framework features a simple policy loss function, which is a reweighted evidence lower bound used in BC-based diffusion policy training and can seamlessly serve as the policy learning module in RL algorithms. To address the exploration challenges arising from the absence of demonstrations, we incorporate reverse curriculum generation and objective-centric representations. Combined with the expressiveness of diffusion policies, our design supports learning of multi-task block-pushing policies in our sparsereward simulation setting. We further evaluate whether the trained diffusion policy transfers in zero-shot to real-world tasks under varying environmental conditions including goal positions, block shapes, block weights and surface friction, providing evidence that this pipeline can transfer to our realworld block-pushing setup under the tested variations. I. INTRODUCTION Denoising diffusion probabilistic model (DDPM) [1], [2],[3] has been showing remarkable expressiveness and flexibility in representing and generating complex data distributions [4] . In light of these strengths, diffusion policies have been widely leveraged to generate long-horizon robot trajectories [5], [6], [7] or imitate expert policies in MDPs [8],[9], [10], [11] using behavior cloning or offline RL methods when expert demonstrations are available in abundance. Building on this foundation, a natural question to ask is whether such expressive policy classes can go beyond singletask learning and serve as a basis for more general-purpose robotic intelligence, since the ultimate goal of autonomous robotic manipulation is to learn one single universal policy that can solve a variety of different tasks [7] . These tasks are often contact-rich, where the presence of frequent and complex contacts makes the planning landscape highly sensitive to differences in both task objectives and external disturbances. Capturing such variability requires sufficiently expressive models to parameterize the universal policies—this is precisely where diffusion policies come into play, owing to their inherent expressiveness and flexibility [12], [13] . Despite the great success of diffusion policies, the current training approaches are generally based on behavior cloning (BC) [6], while RL only serves as an optional fine-tuning method [5] . However, such BC-based approaches could be challenging for multi-task manipulations, since the task space 1Harvard University. 2 Georgia Institute of Technology. requires a lot of diverse expert demonstrations across different tasks, which limits the scalability of the BC approaches. Worse still, the fine-tuning phase is usually fragile and faces the challenge of catastrophic forgetting [?], hindering its ability to reliably adapt to new tasks without compromising the performance of previously learned skills. Recently, a growing interest has emerged in training diffusion policies from scratch using online RL methods [14],[15], [16], which achieves stronger performance compared to previous algorithms based on deterministic or Gaussian policies [17], [18] . In fact, online RL has shown its strong potential for multi-task learning in arcade games and robotics locomotion [19], [20] . However, multi-task diffusion has been studied only with BC-based training [12], [21] but not yet with online RL. Meanwhile, one of the

FAQs

What is the main goal of the proposed method?

Train a single diffusion-policy controller for multi-task, contact-rich block pushing and enable zero-shot transfer from simulation to real-world settings under changing conditions.

How does the paper address the challenge of training without demonstrations?

It incorporates reverse curriculum generation and objective-centric representations to improve exploration when demonstration data is absent.

What factors are used to evaluate zero-shot sim-to-real transfer?

The evaluations vary goal positions, block shapes, block weights, and surface friction to test whether the trained policy transfers to real-world block pushing.

Jul 16, 2026

Upload by Sage