[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-85980-en":3,"doc-seo-85980-105":29,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":11,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},85980,1374391975076,"Riley","https://ap-avatar.wpscdn.com/avatar/14000253ca4ec9f6853?x-image-process=image/resize,m_fixed,w_180,h_180&k=1783305029341752051",8,"Research & Report","Agentic-DPO: From Imitation to Agentic Policy Optimization on Expert Trajectories","Large language model agents are often trained from expert trajectories using supervised fine-tuning (SFT), which imitates token sequences but does not teach state-dependent correction when plausible mistakes arise. Agentic-DPO converts expert demonstrations into lightweight, state-conditioned action preferences using a DPO-style objective with negative one-step actions sampled from the current policy. Policy-Preserving Augmentation keeps the expert policy fixed across schemas. Offline training needs no environment rollouts, reward models, or full-trajectory student exploration, and improves performance across multiple benchmarks.","arXiv :2607 . 1060 1v 1 [ cs .AI] 12 Jul 2026  \nAgentic-DPO: From Imitation to Agentic Policy Optimization on Expert Trajectories  \nYixiong Chen, Alan Yuille  \nJohns Hopkins University  \n[ychen646@jh.edu](ychen646@jh.edu)  \nAbstract  \nLarge Language Model (LLM) agents are commonly trained from expert trajectories using supervised fine-tuning (SFT), which treats multi-turn agent behavior as ordinary text imitation. This recipe is simple and low-cost, but it only learns to imitate the sequence of expert actions, rather than training the agent to choose the right action against plausible mistakes at each state. Existing methods to mitigate this problem include preference learning or reinforcement learning, but they usually need high-cost environment rollouts and reward models. We propose Agentic-DPO, a lightweight offline agent policy optimization method that turns expert trajectories into state-conditioned preference supervision. At each expert action state, Agentic-DPO samples a one-step action from the current state, treats plausible wrong actions as negatives, and contrasts them with the expert action using a DPO-style preference objective. To avoid mixing both policy and schema in preference learning, we introduce Policy-Preserving Augmentation (PPA), which renders the same latent trajectory under multiple schemas while keeping the expert policy fixed. Agentic-DPO requires no online environment rollout, reward model, or full-trajectory student exploration. We conduct experiments across StableToolBench, τ-bench retail, and Mind2Web, where Agentic-DPO consistently improves agents at different model scales beyond imitation. In particular, it raises τ-bench accuracy from 21.7%(SFT) to 41.4% for a 9B model, matching online GRPOunder the same backbone with only step-level rollouts and without environment interaction during gradient steps. The results suggest that expert trajectories can support low-cost agentic policy optimization when converted from demonstrations into state-level action preferences. Code for Agentic-DPO is released at [https://github.com/Schuture/Agentic-DPO](https://github.com/Schuture/Agentic-DPO).  \n1 Introduction  \nLarge Language Model (LLM) agents are increasingly used to solve tasks that require interacting with tools, users, web pages, or software environments over multiple turns [28, 15, 46, 10] . A common way to train such agents is to collect expert trajectories from stronger models or scripted policies, and then apply supervised fine-tuning (SFT) to imitate the full trajectory [21, 6] . This recipe is attractive because it is simple, stable, and does not require online interaction with the environment. However, it inherits a basic limitation of behavior cloning: SFT only imitates the token sequences of the expert trajectory.  \nThis limitation is especially important for agent training, because an agent trajectory is not merely a text response; it is a sequence of state-conditioned decisions. At each step, the model must decide whether to call a tool or respond to the user, which tool or action to choose, and what arguments orgrounding targets to provide. Recent work has started to address this mismatch by showing that policy can be learned better by algorithms that utilizes the structure of expert trajectories. A straightforward  \nPreprint.  \nSFT  \nAgentic-DPO  \nFigure 1: Illustration of the difference between SFT and Agentic-DPO for agent training. SFT treatsan expert trajectory as token-level supervision and trains the model to imitate demonstrated tokens. In contrast, Agentic-DPO treats each expert action as a state-conditioned policy decision. It contrasts the expert action with a one-step action sampled from the current student policy, and trains the model to prefer the expert action over the likely student mistake.  \nway is to improve SFT by selecting only valuable action steps to train, with heuristic rules (SWELego [37]) or the help of external annotation models (ATLaS [6]) . More sophisticate","cbCaiidKXhYZVxTt","https://ap.wps.com/l/cbCaiidKXhYZVxTt","pdf",495424,1,18,"English","en",105,"# Introduction\n## Method Overview\n## Preference Objective and Negative Sampling\n## Policy-Preserving Augmentation\n## Experiments and Results","[{\"question\":\"What problem does Agentic-DPO address compared with supervised fine-tuning (SFT)?\",\"answer\":\"SFT imitates the expert trajectory as token-level text, while Agentic-DPO treats each expert action as a state-conditioned decision and teaches the agent to avoid likely mistakes at each state.\"},{\"question\":\"How does Agentic-DPO build training signals from expert trajectories?\",\"answer\":\"At each expert action state, it samples a one-step action from the current student policy and uses plausible wrong actions as negatives, then forms a preference pair between the expert action and the sampled negative for a DPO-style objective.\"},{\"question\":\"What makes Agentic-DPO lightweight and offline?\",\"answer\":\"Training requires no online environment rollout, no reward-model training, and no full-trajectory student exploration; the rollout stops after the current action and avoids environment interaction during gradient steps.\"}]",1784207540,45,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":27},"agentic-dpo-from-imitation-to-agentic-policy-optimization-on-expert-trajectories","",{"@graph":35,"@context":85},[36,53,68],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/agentic-dpo-from-imitation-to-agentic-policy-optimization-on-expert-trajectories/85980/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-25","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":11},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What problem does Agentic-DPO address compared with supervised fine-tuning (SFT)?","Question",{"text":75,"@type":76},"SFT imitates the expert trajectory as token-level text, while Agentic-DPO treats each expert action as a state-conditioned decision and teaches the agent to avoid likely mistakes at each state.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"How does Agentic-DPO build training signals from expert trajectories?",{"text":80,"@type":76},"At each expert action state, it samples a one-step action from the current student policy and uses plausible wrong actions as negatives, then forms a preference pair between the expert action and the sampled negative for a DPO-style objective.",{"name":82,"@type":73,"acceptedAnswer":83},"What makes Agentic-DPO lightweight and offline?",{"text":84,"@type":76},"Training requires no online environment rollout, no reward-model training, and no full-trajectory student exploration; the rollout stops after the current action and avoids environment interaction during gradient steps.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":45,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":45,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":45,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":45,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":45,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":45,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":45,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]