[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-85652-en":3,"doc-seo-85652-105":29,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},85652,4398048949847,"Eliana","https://ap-avatar.wpscdn.com/avatar/400002536579ef2da7f?_k=1778318612642679267",8,"Research & Report","Freeform Preference Learning for Robotic Manipulation","Reward design is a key bottleneck in improving autonomous robots, especially in long-horizon manipulation where sparse success signals provide weak learning gradients and binary preference labels merge distinct notions of quality into an ambiguous outcome. Freeform Preference Learning (FPL) learns robot policies from human freeform preferences by letting annotators define natural-language preference axes and give per-axis pairwise comparisons. A language-conditioned reward model enables dense axis-specific feedback, and a reward-conditioned policy optimizes across multiple dimensions, outperforming sparse and binary methods by 38 percentage points.","arXiv :2606 .32027v2 [ cs .RO] 13 Jul 2026  \nFreeform Preference Learning for Robotic  \nManipulation  \nMarcel Torne∗ Anubha Mahajan∗ Abhijnya Bhat∗ Chelsea Finn  \nStanford University  \nAbstract: Reward design remains a central bottleneck for autonomous robot policy improvement, especially in long-horizon manipulation tasks where sparse success labels provide too little signal and binary preferences collapse many competing notions of quality into one ambiguous signal. We introduce Freeform Preference Learning (FPL), a method for learning robot policies from freeform human preferences. Rather than asking annotators which of two trajectories is better overall, FPL lets them define natural-language preference axes, such as speed, safety, quality of placement, or carefulness, and provide pairwise preferences along each axis. These annotations are used to learn a language-conditioned reward model that maps a trajectory and preference label to an axis-specific reward.  \nWe use this model to train a reward-conditioned policy that optimizes across the multiple human-specified dimensions. Across four real-world and two simulated long-horizon manipulation tasks, FPL improves over sparse-reward and binarypreference methods by 38 percentage points. Beyond improved performance, FPL learns dense progress signals without explicit subtask segmentation, shows compositionality of behavior not present in the data, and allows users to steer the policy towards different behaviors at test time without retraining. Blog post with videos available [at](at freeform-pl.github.io/fpl.website/)[ freeform-pl.github.io/fpl.website/](at freeform-pl.github.io/fpl.website/)  \nKeywords: Reinforcement Learning, Preference Learning, Robot Manipulation  \n1 Introduction  \nRewards are a critical component for autonomous robot improvement. An ideal reward function should provide dense, unambiguous feedback and should capture all aspects of desirable behavior. For example, supervision for the simple task of setting a table should incorporate the configuration of the cutlery, the degree of care taken to not break fragile plates, the comfort of nearby people (e.g. to avoid motions that point a knife towards a person), and the speed of execution, among other aspects. Accurately capturing all of these axes presents a major challenge, both when eliciting supervision from people and when representing all of these factors in a reward function and downstream behavior. Moreover, a reward function that captures these axes but is too sparse, or dense but inaccurate, can lead to unwanted downstream behaviors when optimized against. In this paper, we study how to leverage human supervision to learn reward functions and ultimately robot behavior that captures all dimensions of a person’s intent.  \nPrior works have studied a variety of rewards and reward learning approaches. Perhaps the simplest option is to provide or learn from binary success labels [1, 2, 3], which should in principle make it easy for people to determine if all criteria are met. However, this reward signal places significant burden on the reinforcement learning algorithm, making it hard to scale to more challenging tasks and to incorporate real-world constraints on behavior beyond basic task completion. Other works learn shaped scalar rewards [4, 5, 6] but still focus on task progress metrics, ignoring important criteria on how a task was performed. Finally, preference learning [7, 8, 9] is a promising paradigm  \n∗Equal contribution  \nFigure 1: Comparison between binary preference learning (left) and Freeform Preference Learning (right) . Binary preference learning asks annotators to compare two trajectories using a single “overall preference”. Freeform Preference Learning instead collects more detailed feedback by letting the annotator specify the axes in natural-language and provide per-axis preferences.  \nfor learning denser reward signals while reducing the burden on human supervisors, but requires them to col","cbCailnRQIpeyIDI","https://ap.wps.com/l/cbCailnRQIpeyIDI","pdf",9975183,1,24,"English","en",105,"# Introduction\n## Reward design challenges in long-horizon manipulation\n## Prior reward learning and preference learning limitations\n## Core idea and approach of FPL","[{\"question\":\"Why is reward design difficult for long-horizon robotic manipulation?\",\"answer\":\"Long-horizon tasks often provide sparse success labels, giving insufficient learning signal. Binary preferences can also collapse multiple quality dimensions into one ambiguous label.\"},{\"question\":\"How does Freeform Preference Learning collect supervision from humans?\",\"answer\":\"FPL asks annotators to define relevant natural-language preference axes (e.g., speed, safety, placement quality) and provide pairwise preferences for each axis.\"},{\"question\":\"What improvements does FPL achieve compared with sparse-reward and binary preference methods?\",\"answer\":\"Across four real-world and two simulated long-horizon manipulation tasks, policies trained with FPL improve over sparse-reward and binary preference methods by 38 percentage points, while also providing dense progress signals and test-time steerability.\"}]",1784205371,60,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":27},"freeform-preference-learning-for-robotic-manipulation","",{"@graph":35,"@context":85},[36,53,68],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/freeform-preference-learning-for-robotic-manipulation/85652/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-17","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"Why is reward design difficult for long-horizon robotic manipulation?","Question",{"text":75,"@type":76},"Long-horizon tasks often provide sparse success labels, giving insufficient learning signal. Binary preferences can also collapse multiple quality dimensions into one ambiguous label.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"How does Freeform Preference Learning collect supervision from humans?",{"text":80,"@type":76},"FPL asks annotators to define relevant natural-language preference axes (e.g., speed, safety, placement quality) and provide pairwise preferences for each axis.",{"name":82,"@type":73,"acceptedAnswer":83},"What improvements does FPL achieve compared with sparse-reward and binary preference methods?",{"text":84,"@type":76},"Across four real-world and two simulated long-horizon manipulation tasks, policies trained with FPL improve over sparse-reward and binary preference methods by 38 percentage points, while also providing dense progress signals and test-time steerability.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,109,114,119,122,127,130,134],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":45,"category_name":107,"show_sort_weight":28,"slug":108},5,"Comic","comic",{"id":110,"doc_module":4,"doc_module_name":45,"category_name":111,"show_sort_weight":112,"slug":113},6,"Technology",50,"technology",{"id":115,"doc_module":4,"doc_module_name":45,"category_name":116,"show_sort_weight":117,"slug":118},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":120,"slug":121},30,"research-report",{"id":123,"doc_module":4,"doc_module_name":45,"category_name":124,"show_sort_weight":125,"slug":126},9,"Religion & Spirituality",20,"religion-spirituality",{"id":125,"doc_module":4,"doc_module_name":45,"category_name":128,"show_sort_weight":125,"slug":129},"World Cup","world-cup",{"id":131,"doc_module":4,"doc_module_name":45,"category_name":132,"show_sort_weight":131,"slug":133},10,"Lifestyle","lifestyle",{"id":135,"doc_module":4,"doc_module_name":45,"category_name":136,"show_sort_weight":106,"slug":137},19,"General","general"]