[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"detail-sidebar-cat-0-en-105":3,"doc-seo-501572-105":59,"doc-detail-501572-en":130},{"code":4,"msg":5,"data":6},0,"success",[7,13,18,23,28,33,38,43,48,51,55],{"id":8,"doc_module":4,"doc_module_name":9,"category_name":10,"show_sort_weight":11,"slug":12},1,"Document","Story & Novel",90,"story-novel",{"id":14,"doc_module":4,"doc_module_name":9,"category_name":15,"show_sort_weight":16,"slug":17},2,"Literature",80,"literature",{"id":19,"doc_module":4,"doc_module_name":9,"category_name":20,"show_sort_weight":21,"slug":22},4,"Exam",70,"exam",{"id":24,"doc_module":4,"doc_module_name":9,"category_name":25,"show_sort_weight":26,"slug":27},5,"Comic",60,"comic",{"id":29,"doc_module":4,"doc_module_name":9,"category_name":30,"show_sort_weight":31,"slug":32},6,"Technology",50,"technology",{"id":34,"doc_module":4,"doc_module_name":9,"category_name":35,"show_sort_weight":36,"slug":37},7,"Healthcare",40,"healthcare",{"id":39,"doc_module":4,"doc_module_name":9,"category_name":40,"show_sort_weight":41,"slug":42},8,"Research & Report",30,"research-report",{"id":44,"doc_module":4,"doc_module_name":9,"category_name":45,"show_sort_weight":46,"slug":47},9,"Religion & Spirituality",20,"religion-spirituality",{"id":46,"doc_module":4,"doc_module_name":9,"category_name":49,"show_sort_weight":46,"slug":50},"World Cup","world-cup",{"id":52,"doc_module":4,"doc_module_name":9,"category_name":53,"show_sort_weight":52,"slug":54},10,"Lifestyle","lifestyle",{"id":56,"doc_module":4,"doc_module_name":9,"category_name":57,"show_sort_weight":24,"slug":58},19,"General","general",{"code":4,"msg":60,"data":61},"ok",{"site_id":62,"language":63,"slug":64,"title":65,"keywords":66,"description":67,"schema_data":68,"social_meta":123,"head_meta":125,"extra_data":127,"updated_unix":129},105,"en","aligning-language-models-with-demonstrated-feedback-ditto","ALIGNING LANGUAGE MODELS WITH DEMONSTRATED FEEDBACK - - DITTO","","Language models are aligned to imitate a collective voice, often producing outputs that fit no particular user or setting. Steering via supervised finetuning or RLHF is effective but demands very large datasets for new ad-hoc tasks. The proposed approach aligns an LLM to a specific context using fewer than ten demonstrations as feedback. Demonstration ITerated Task Optimization (DITTO) builds preference pairs by treating user demonstrations as preferred over LLM outputs and earlier checkpoints, then trains with preference optimization such as DPO. Experiments and a user study show higher win rates than few-shot prompting, supervised fine-tuning, and related self-play methods, with an average gain of 19 percentage points.",{"@graph":69,"@context":122},[70,84,105],{"@type":71,"itemListElement":72},"BreadcrumbList",[73,77,79,82],{"item":74,"name":75,"@type":76,"position":8},"https://docshare.wps.com","Home","ListItem",{"item":78,"name":9,"@type":76,"position":14},"https://docshare.wps.com/document/",{"item":80,"name":30,"@type":76,"position":81},"https://docshare.wps.com/document/technology/",3,{"item":83,"name":65,"@type":76,"position":19},"https://docshare.wps.com/document/aligning-language-models-with-demonstrated-feedback-ditto/501572/",{"url":83,"name":65,"@type":85,"image":86,"author":91,"headline":65,"publisher":94,"fileFormat":97,"inLanguage":63,"description":67,"dateModified":98,"datePublished":99,"encodingFormat":97,"isAccessibleForFree":100,"interactionStatistic":101},"DigitalDocument",{"url":87,"@type":88,"width":89,"height":90},"https://docshare.wps.com/thumbnails/aligning-language-models-with-demonstrated-feedback-ditto/501572.png","ImageObject",300,407,{"name":92,"@type":93},"Indoniesian Boy","Person",{"url":74,"name":95,"@type":96},"DocShare","Organization","application/pdf","2026-10-16","2026-10-03",true,{"@type":102,"interactionType":103,"userInteractionCount":29},"InteractionCounter",{"@type":104},"ViewAction",{"@type":106,"mainEntity":107},"FAQPage",[108,114,118],{"name":109,"@type":110,"acceptedAnswer":111},"Why do general-purpose language models often produce generic outputs?","Question",{"text":112,"@type":113},"They are trained to reflect a broad, universal style, which can mismatch a user’s specific task, writing style, or audience needs.","Answer",{"name":115,"@type":110,"acceptedAnswer":116},"How does DITTO use a small number of demonstrations to align an LLM?",{"text":117,"@type":113},"DITTO treats user demonstrations as preferred over the LLM’s outputs and even earlier training checkpoints, iteratively constructing preference comparisons for training.",{"name":119,"@type":110,"acceptedAnswer":120},"What evidence supports DITTO’s effectiveness?",{"text":121,"@type":113},"The paper reports evaluations across domains (news, emails, blog posts) and a user study (N=16), where DITTO outperforms few-shot prompting, supervised fine-tuning, and other self-play methods by an average of 19 percentage points.","https://schema.org",{"og:url":83,"og:type":124,"og:title":65,"og:site_name":95,"og:description":67},"article",{"robots":126,"canonical":83},"index,follow",{"doc_id":128,"site_id":62},501572,1791025875,{"code":4,"msg":5,"data":131},{"doc_id":128,"user_id":132,"nickname":92,"user_avatar":133,"doc_module":4,"category_id":29,"category_name":30,"doc_title":65,"doc_description":67,"doc_content":134,"file_id":135,"file_url":136,"file_type":137,"file_size":138,"view_count":29,"is_deleted":4,"is_public":8,"is_downloadable":8,"audit_status":8,"page_count":139,"language":140,"language_code":63,"site_id":62,"html_lang":63,"table_of_contents":141,"faqs":142,"seo_title":143,"seo_description":67,"update_tm":144,"read_time":145},962090760608,"https://ap-avatar.wpscdn.com/davatar_276721f389ce27ea32af1340a28f341c","ALIGNING LANGUAGE MODELS WITH DEMONSTRATED FEEDBACK  \nOmar Shaikh∗  \nStanford University [oshaikh@stanford.edu](oshaikh@stanford.edu)  \nMichelle S. Lam∗ Stanford University [mlam4@cs.stanford.edu](mlam4@cs.stanford.edu)  \nJoey Hejna∗  \nStanford University [jhejna@cs.stanford.edu](jhejna@cs.stanford.edu)  \narXiv :2406 .00888v2 [ cs .CL] 18 Apr 2025  \nYijia Shao Hyundong Cho Michael S. Bernstein Diyi Yang  \nStanford University USC Stanford University Stanford University  \nABSTRACT  \nLanguage models are aligned to emulate the collective voice of many, resulting in outputs that align with no one in particular. Steering LLMs away from generic output is possible through supervised finetuning or RLHF, but requires prohibitively large datasets for new ad-hoc tasks. We argue that it is instead possible to align an LLM to a specific setting by leveraging a very small number ( \u003C 10) of demonstrations as feedback. Our method, Demonstration ITerated Task Optimization (DITTO), directly aligns language model outputs to a user’s demonstrated behaviors. Derived using ideas from online imitation learning, DITTO cheaply generates online comparison data by treating users’ demonstrations as preferred over output from the LLM and its intermediate checkpoints. Concretely, DITTO operates by having an LLM generate examples that are presumed to be inferior to expert demonstrations. The method iteratively constructs pairwise preference relationships between these LLM-generated samples and expert demonstrations, potentially including comparisons between different training checkpoints. These constructed preference pairs are then used to train the model using a preference optimization algorithm (e.g. DPO) . We evaluate DITTO’s ability to learn fine-grained style and task alignment across domains such as news articles, emails, and blog posts.  \nAdditionally, we conduct a user study soliciting a range of demonstrations from participants (N = 16) . Across our benchmarks and user study, we find that winrates for DITTO outperform few-shot prompting, supervised fine-tuning, and other self-play methods by an avg. of 19% points. By using demonstrations as feedback directly, DITTO offers a novel method for effective customization of LLMs.1  \n1 INTRODUCTION  \nLarge language models (LLMs) are trained for general-purpose use. In practice, however, they are often applied to very specific tasks for very specific users. Consider a task as simple as writing an email: our preferred email depends on personal writing style, the specific email task, or the target audience (a friend, stranger, etc.) . As a result, there can be a mismatch between the universal style (Santurkar et al., 2023; Chakrabarty et al., 2023) trained into an LLM via instruction and preference tuning, and the specific style needed for applications. LLM outputs feel unopinionated and generic because of this mismatch.  \nWhile existing approaches such as supervised or preference finetuning are effective, they can require a large corpus of (un)acceptable behavior (on the order of ≈ 1K samples (Zhou et al., 2024; Ouyang et al., 2022)), which in turn requires unreasonably high effort from an individual. RLAIF methods like Constitutional AI (Bai et al., 2022) automate pairwise preference collection with an LLM, but align models to general principles that may not capture fine-grained preferences. Although prompting is data efficient, finding an effective prompt can be tedious—end-users often rely on brittle prompting heuristics (Zhou et al., 2022; Zamfirescu-Pereira et al., 2023) . How might we efficiently communicate preferences and align a language model to a new individual or task?  \n∗Equal Contribution  \n1Code: [https://github.com/SALT-NLP/demonstrated-feedback](https://github.com/SALT-NLP/demonstrated-feedback)  \nFigure 1: DITTO iteratively aligns LLMs to demonstrated behavior. When a user supplies demonstrations (through edits to a model’s output, past preferred interaction history, or writing examples from","cbCaig9X9pY1jxi1","https://ap.wps.com/l/cbCaig9X9pY1jxi1","pdf",1159818,28,"English","# Abstract\n# Introduction\n## Background and Motivation\n## DITTO Framework\n## Benchmarking and Evaluation","[{\"question\":\"Why do general-purpose language models often produce generic outputs?\",\"answer\":\"They are trained to reflect a broad, universal style, which can mismatch a user’s specific task, writing style, or audience needs.\"},{\"question\":\"How does DITTO use a small number of demonstrations to align an LLM?\",\"answer\":\"DITTO treats user demonstrations as preferred over the LLM’s outputs and even earlier training checkpoints, iteratively constructing preference comparisons for training.\"},{\"question\":\"What evidence supports DITTO’s effectiveness?\",\"answer\":\"The paper reports evaluations across domains (news, emails, blog posts) and a user study (N=16), where DITTO outperforms few-shot prompting, supervised fine-tuning, and other self-play methods by an average of 19 percentage points.\"}]","ALIGNING LANGUAGE MODELS WITH DEMONSTRATED FEEDBACK - - DITTO | PDF",1791020812,71]