[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-85027-en":3,"doc-seo-85027-105":29,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},85027,13056703019404,"Miles","https://ap-avatar.wpscdn.com/davatar_29158cc5080c5b710cf443261637dec0",8,"Research & Report","Selective Timestep Weighting and Advantage-Based Replay for Sample-Efficient Diffusion RLHF","Reinforcement learning from human feedback (RLHF) can align generative diffusion models with human preferences, but diffusion RLHF is highly feedback-inefficient because learning depends on expensive human or reward-model evaluations. This work introduces two complementary strategies to improve feedback efficiency while maintaining generalization to unseen prompts. It leverages the observation that reward signal across denoising timesteps and trajectories is uneven, enabling more informative gradient updates. The paper presents per-timestep weighting and advantage-based trajectory replay, achieving up to 6× better sample efficiency under matched hyperparameters.","Selective Timestep Weighting and Advantage-Based Replay for Sample-Efficient  \nDiffusion RLHF  \nEric Zhu Carnegie Mellon University  \n[ezhu3@andrew.cmu.edu](ezhu3@andrew.cmu.edu)  \nAbhinav Shrivastava University of Maryland, College Park  \n[abhinav2@umd.edu](abhinav2@umd.edu)  \nSoumik Mukhopadhyay University of Maryland, College Park  \n[soumik@umd.edu](soumik@umd.edu)  \narXiv :2607 .07693v 1 [ cs .LG] 8 Jul 2026  \nAbstract  \nReinforcement learning from human feedback (RLHF) has emerged as a powerful paradigm for aligning generative models with human preferences. However, applying RLHF to diffusion models remains highly feedback inefficient, as existing approaches typically require large amounts of human or reward model evaluations. This limitation reduces the practicality of diffusion RLHF in realworld settings where feedback is the primary bottleneck. In this paper, we propose two complementary strategies that substantially improve the feedback efficiency of diffusion RLHF while preserving generalization to unseen prompts. Our key observation is that reward information in diffusion trajectories is unevenly distributed: not all denoising timesteps or trajectories contribute equally to learning from a reward signal. By emphasizing informative timesteps and trajectories during optimization, we obtain more effective gradient updates. First, we introduce a per-timestep weighting scheme that reweights denoising steps during policy optimization. We theoretically connect this weighting to the optimal convergence properties of proximal policy optimization (PPO) and approximate the resulting weighting trend empirically. Second, we introduce a replay mechanism that prioritizes informative trajectories, enabling the model to reuse past samples instead of repeatedly querying new rewards. Together, these strategies significantly improve the feedback efficiency of diffusion RLHF. Under identical hyperparameter settings, our approach achievesup to a 6 × improvement in sample efficiency compared to widely used diffusion RLHF baselines.  \n1. Introduction  \nDiffusion models [41] have become the leading framework for high-quality image generation. However, because they  \nFigure 1 . Our method has two parts. In the per-timestep weighting, we consider the relative importance of each transition step in the denoising trajectory. In the historical hardmining part, we look at previous trajectories with high advantage and repeat those trajectories in the training.  \nFigure 2 . A graph of a baseline method (bottom curve) vs. the same baseline method using our method (top curve) . As shown above, augmenting existing RLHF frameworks with our method achieves significant sample efficiency compared to the baseline.  \nare trained to reproduce the distribution of their training data, they do not inherently reflect human preferences. Recent work [2, 8] addresses this limitation through reinforcement learning from human feedback (RLHF) [6], which fine-tunes diffusion models using scalar feedback from human or reward models to explicitly optimize for preference alignment.  \nOne major challenge with RLHF in diffusion models is the credit assignment problem: the difficulty of determining which intermediate timesteps in the diffusion process actually contributed to the final reward. Because human feedback is only given on the final image, methods such as DDPO [2] simply assign the same loss to every timestep. This setup ignores the structure of the denoising process and  \nhow different timesteps edit the image at different levels of granularity [25] . As a result, the model optimizes transitions uniformly without considering the non-uniform nature of the denoising trajectory, leading to inefficient training.  \nPrior work attempts to address this issue by contrasting paired denoising trajectories generated from the same initial noise [14, 52] . In these approaches, trajectories remain identical until a designated branching timestep and diverge afterward. While contra","cbCaislmAhtLPgB7","https://ap.wps.com/l/cbCaislmAhtLPgB7","pdf",27358478,1,19,"English","en",105,"# Introduction\n## Per-timestep weighting scheme\n## Advantage-based trajectory replay mechanism","[{\"question\":\"Why is RLHF feedback inefficient for diffusion models?\",\"answer\":\"RLHF relies on human or reward-model feedback, but existing diffusion RLHF methods effectively treat all denoising timesteps as equally contributing to the final reward. This causes inefficient credit assignment and requires many evaluations to learn useful behavior.\"},{\"question\":\"What is the proposed per-timestep weighting strategy?\",\"answer\":\"The method reweights denoising steps during policy optimization so more informative timesteps receive higher weight. It is theoretically related to PPO advantage variance and approximated using a heuristic based on per-timestep latent change.\"},{\"question\":\"How does advantage-based replay improve diffusion RLHF training?\",\"answer\":\"Instead of discarding previously sampled trajectories, the approach reuses informative ones by prioritizing trajectories with high advantage. This reduces the need for repeatedly querying new rewards and increases sample efficiency.\"}]",1784200483,48,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":27},"selective-timestep-weighting-and-advantage-based-replay-for-sample-efficient-diffusion-rlhf","",{"@graph":35,"@context":85},[36,53,68],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/selective-timestep-weighting-and-advantage-based-replay-for-sample-efficient-diffusion-rlhf/85027/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-17","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"Why is RLHF feedback inefficient for diffusion models?","Question",{"text":75,"@type":76},"RLHF relies on human or reward-model feedback, but existing diffusion RLHF methods effectively treat all denoising timesteps as equally contributing to the final reward. This causes inefficient credit assignment and requires many evaluations to learn useful behavior.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"What is the proposed per-timestep weighting strategy?",{"text":80,"@type":76},"The method reweights denoising steps during policy optimization so more informative timesteps receive higher weight. It is theoretically related to PPO advantage variance and approximated using a heuristic based on per-timestep latent change.",{"name":82,"@type":73,"acceptedAnswer":83},"How does advantage-based replay improve diffusion RLHF training?",{"text":84,"@type":76},"Instead of discarding previously sampled trajectories, the approach reuses informative ones by prioritizing trajectories with high advantage. This reduces the need for repeatedly querying new rewards and increases sample efficiency.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":45,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":45,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":45,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":45,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":45,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":45,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":21,"doc_module":4,"doc_module_name":45,"category_name":136,"show_sort_weight":106,"slug":137},"General","general"]