[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-81638-en":3,"doc-seo-81638-105":29,"detail-sidebar-cat-0-en-105":90},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":4,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},81638,8796095360427,"Lucas Martin","https://ap-avatar.wpscdn.com/davatar_994ba38a5ba835b3df7d355c54d3ed8d",8,"Research & Report","HiPO: Hierarchical Preference Optimization for Adaptive Reasoning in Large Language Models","Direct Preference Optimization (DPO) aligns large language models with human preferences but struggles on complex reasoning, since it optimizes the full response as one unit and cannot provide feedback for individual parts of multi-step solutions. HiPO (Hierarchical Preference Optimization) extends DPO by decomposing responses into reasoning segments—query clarification and context, reasoning steps, and the final answer—and computing a weighted DPO loss for each segment. HiPO keeps DPO’s computational efficiency while enabling segment-specific training. Experiments on Math Stack Exchange with multiple 7B LLMs show higher performance on math benchmarks and improved organization, logical flow, and consistency measured by GPT-4.1.","HIPO: HIERARCHICAL PREFERENCE OPTIMIZATION FOR ADAPTIVE REASONING IN LARGE LANGUAGE MODELS  \nDarsh Kachroo  \nVellore Institute of Technology, Chennai [darsh.kachroo2022@vitstudent.ac.in](darsh.kachroo2022@vitstudent.ac.in)  \nAdriana Caraeni  \nUniversity of Massachusetts Amherst  \n[acaraeni@umass.edu](acaraeni@umass.edu)  \nArjun Prasaath Anbazhagan  \nNorthwestern University [arjunanbazhagan2026@u.northwestern.edu](arjunanbazhagan2026@u.northwestern.edu)  \narXiv :2604 .20 140v2 [ cs .AI] 10 Jul 2026  \nBrennan Lagasse  \nYale University / Algoverse AI Research [brennan@algoverseairesearch.org](brennan@algoverseairesearch.org)  \nKevin Zhu  \nAlgoverse AI Research [kevin@algoverse.us](kevin@algoverse.us)  \nABSTRACT  \nDirect Preference Optimization (DPO) is an effective framework for aligning large language models with human preferences, but it struggles with complex reasoning tasks. DPO optimizes for the likelihood of generating preferred over dispreferred responses in their entirety and lacks the granularity to provide feedback on subsections of many-step solutions typical of reasoning tasks. Existing methods excel at either stable preference learning (e.g., DPO variants like KTO and RSO) or structured reasoning (e.g., ReMA’s multi-agent RL framework, Tree of Thoughts), but fail to merge these complementary strengths. We propose HiPO (Hierarchical Preference Optimization), an extension of DPO that separates responses into reasoning segments (query clarification and context, reasoning steps, and answer) and computes loss as a weighted sum of the DPO loss for each segment.  \nOur approach enables segment-specific training while maintaining DPO’s computational efficiency and training stability. We demonstrate that for multiple 7B LLMs fine-tuned using HiPO and DPO on the Math Stack Exchange preference dataset, the models trained with HiPO outperform the others on a variety of common math benchmarks and achieve greater organization, logical flow, and consistency as measured by GPT-4.1 .  \nKeywords Preference Optimization · Reasoning · Large Language Models · Direct Preference Optimization · Hierarchical Learning  \n1 Introduction  \n1.1 Motivation  \nDirect Preference Optimization (DPO) efficiently aligns large language models with human preferences by directly optimizing pairwise preference data, reducing required computation costs. This method is an alternative to reinforcement learning approaches, which are extremely resource-intensive. However, DPO’s monolithic treatment of responses—lacking component-level granularity—creates a fundamental limitation: it cannot separately optimize different reasoning components (query clarification, structured reasoning, and answer formulation), preventing practitioners from targeting specific deficiencies or adapting training emphasis to problem-specific reasoning demands.  \nDifferent reasoning tasks naturally require different cognitive skills. A problem with an unclear or ambiguous question demands careful query interpretation prior to any solution attempt. Complex, multi-step problems requiring significant domain knowledge benefit from structured meta-cognitive reasoning: the ability to plan, organize steps, and apply  \nrelevant concepts systematically. Additionally, tasks requiring specific output formats, such as competition submissions or standardized test responses, require precise final answer formulation. Standard DPO optimizes all aspects as one, and does not natively support targeted improvement of specific reasoning deficiencies.  \nHiPO addresses this limitation by decomposing responses into distinct reasoning segments and enabling practitioners to control training emphasis through adjustable segment weights. This allows prioritizing query understanding for problems with ambiguous specifications, strengthening step-by-step reasoning for complex analytical tasks, or building complex reasoning capabilities incrementally by first developing query interpretation, then meta-cognitive planning, ","cbCaira6bfgKugXZ","https://ap.wps.com/l/cbCaira6bfgKugXZ","pdf",330351,1,11,"English","en",105,"# Abstract\n# Introduction\n## Motivation\n## Applications\n## Contributions","[{\"question\":\"What problem does DPO face on complex reasoning tasks?\",\"answer\":\"DPO treats the entire response as a single unit and lacks component-level granularity, so it cannot independently optimize different reasoning parts such as query clarification, step-by-step reasoning, and final answer formulation.\"},{\"question\":\"How does HiPO differ from DPO in training?\",\"answer\":\"HiPO decomposes each response into reasoning segments—query clarification and context, reasoning steps, and answer—then computes loss as a weighted sum of DPO loss for each segment, enabling segment-specific training emphasis.\"},{\"question\":\"What benefits does HiPO provide in experiments with 7B LLMs?\",\"answer\":\"Models fine-tuned with HiPO outperform DPO-based counterparts on common math benchmarks and show better organization, logical flow, and consistency as measured by GPT-4.1.\"}]",1784175038,28,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":85,"head_meta":87,"extra_data":89,"updated_unix":27},"hipo-hierarchical-preference-optimization-for-adaptive-reasoning-in-large-language-models","",{"@graph":35,"@context":84},[36,53,67],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/hipo-hierarchical-preference-optimization-for-adaptive-reasoning-in-large-language-models/81638/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":61,"encodingFormat":60,"isAccessibleForFree":62,"interactionStatistic":63},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-16",true,{"@type":64,"interactionType":65,"userInteractionCount":4},"InteractionCounter",{"@type":66},"ViewAction",{"@type":68,"mainEntity":69},"FAQPage",[70,76,80],{"name":71,"@type":72,"acceptedAnswer":73},"What problem does DPO face on complex reasoning tasks?","Question",{"text":74,"@type":75},"DPO treats the entire response as a single unit and lacks component-level granularity, so it cannot independently optimize different reasoning parts such as query clarification, step-by-step reasoning, and final answer formulation.","Answer",{"name":77,"@type":72,"acceptedAnswer":78},"How does HiPO differ from DPO in training?",{"text":79,"@type":75},"HiPO decomposes each response into reasoning segments—query clarification and context, reasoning steps, and answer—then computes loss as a weighted sum of DPO loss for each segment, enabling segment-specific training emphasis.",{"name":81,"@type":72,"acceptedAnswer":82},"What benefits does HiPO provide in experiments with 7B LLMs?",{"text":83,"@type":75},"Models fine-tuned with HiPO outperform DPO-based counterparts on common math benchmarks and show better organization, logical flow, and consistency as measured by GPT-4.1.","https://schema.org",{"og:url":51,"og:type":86,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":88,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":91},[92,96,100,104,109,114,119,122,127,130,134],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":93,"show_sort_weight":94,"slug":95},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":97,"show_sort_weight":98,"slug":99},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":101,"show_sort_weight":102,"slug":103},"Exam",70,"exam",{"id":105,"doc_module":4,"doc_module_name":45,"category_name":106,"show_sort_weight":107,"slug":108},5,"Comic",60,"comic",{"id":110,"doc_module":4,"doc_module_name":45,"category_name":111,"show_sort_weight":112,"slug":113},6,"Technology",50,"technology",{"id":115,"doc_module":4,"doc_module_name":45,"category_name":116,"show_sort_weight":117,"slug":118},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":120,"slug":121},30,"research-report",{"id":123,"doc_module":4,"doc_module_name":45,"category_name":124,"show_sort_weight":125,"slug":126},9,"Religion & Spirituality",20,"religion-spirituality",{"id":125,"doc_module":4,"doc_module_name":45,"category_name":128,"show_sort_weight":125,"slug":129},"World Cup","world-cup",{"id":131,"doc_module":4,"doc_module_name":45,"category_name":132,"show_sort_weight":131,"slug":133},10,"Lifestyle","lifestyle",{"id":135,"doc_module":4,"doc_module_name":45,"category_name":136,"show_sort_weight":105,"slug":137},19,"General","general"]