[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-81497-en":3,"doc-seo-81497-105":29,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},81497,1099513958607,"Jiven","https://ap-avatar.wpscdn.com/avatar/100002390cf8733938c?x-image-process=image/resize,m_fixed,w_180,h_180&k=1778829742770036399",8,"Research & Report","Evaluating Task-Oriented Dialogue Consistency through Constraint Satisfaction","Task-oriented dialogue systems require consistency internally across turns and externally with the domain knowledge represented by a conversational knowledge base. The work models dialogue consistency as a Constraint Satisfaction Problem (CSP): dialogue segments become variables and linguistic, conversational, and domain-based properties form constraints. A CSP solver is used to detect inconsistencies in dialogues re-lexicalized by an LLM, showing CSP’s effectiveness and revealing substantial accuracy limits for current models. An ablation study highlights domain-knowledge constraints as the main source of violations, and argues CSP captures key properties beyond pipeline-based methods.","Evaluating Task-Oriented Dialogue Consistency through Constraint  \nSatisfaction  \nTiziano Labruna and Bernardo Magnini  \nFondazione Bruno Kessler  \nVia Sommarive, 18, Trento, Italy  \narXiv :2407 . 1 1857v 1 [ cs .CL] 16 Jul 2024  \nAbstract  \nTask-oriented dialogues must maintain consistency both within the dialogue itself, ensuring logical coherence across turns, and with the conversational domain, accurately reflecting external knowledge. We propose to conceptualize dialogue consistency as a Constraint Satisfaction Problem (CSP), wherein variables represent segments of the dialogue referencing the conversational domain, and constraints among variables reflect dialogue properties, including linguistic, conversational, and domain-based aspects. To demonstrate the feasibility of the approach, we utilize a CSP solver to detect inconsistencies in dialogues re-lexicalized by an  \nLLM. Our findings indicate that: (i) CSP is effective to detect dialogue inconsistencies; and (ii) consistent dialogue re-lexicalization is challenging for state-of-the-art LLMs, achieving only a 0.15 accuracy rate when compared to a CSP solver. Furthermore, through an ablation study, we reveal that constraints derived from domain knowledge pose the greatest difficulty in being respected. We argue that CSP captures core properties of dialogue consistency that have been poorly considered by approaches based on component pipelines.  \n1 Introduction  \nTask-oriented dialogue (TOD) systems (McTear, 2020 ; Louvan and Magnini, 2020 ; Balaraman et al., 2021) play a crucial role in human-computer interaction, facilitating seamless communication between users and machines to accomplish specific tasks. A peculiar characteristic of TODs is that they need to maintain consistency at two levels: (i) internally within the dialogue itself, ensuring that information in a turn is consistent with information in other turns, and (ii) consistency with the content of the conversational domain of the dialogue system. Internal consistency is responsible for the coherence of the dialogue, making it possible to maintain a meaningful exchange between  \nR1: N=Taberna A=centre F=spanish P=cheap R2: N=Espana A=centre F=spanish P=moderate R3: N=Beirut A=centre F=lebanese P=cheap  \nU 1: I am looking for a restaurant serving Spanish food.  \nS 1: There are three restaurants serving Spanish food, one is cheap and the other is moderate price range. Which price range would you prefer?  \nU 2: I am looking for a cheap restaurant in any area that serves Spanish food.  \nS 2: Beirut is cheap and serves Lebanese food. Would you like the location information?  \nFigure 1: An inconsistent task-oriented dialogue with a Knowledge Base. Red values indicate internal inconsistencies, purple values indicate external inconsistencies.  \nthe participants. External consistency, on the other hand, allows the dialogue to correctly reflect domain knowledge. In this paper, we investigate how dialogue consistency in TOD can be effectively modeled such that possible violations (i.e., inconsistencies) can be automatically detected.  \nFigure 1, shows a fragment of a Knowledge Base (three restaurants in a city) and a short dialogue in which a user expresses preferences for restaurants serving Spanish food, and the system responds providing information about available options. There are two inconsistencies in this dialogue: first, at turn S1, the system mentions three restaurants serving Spanish food, which is not consistent with the domain knowledge, where there are two such restaurants (domain inconsistency) . Second, at turn S2, the system introduces a Lebanese restaurant, while it would have been expected to mention a  \nSpanish restaurant (dialogue inconsistency) . We assume that a well-formed TOD should not manifest any inconsistency of the type reported in our example. However, while relevant work on evaluating TODs has focused on single dialogue components (e.g., dialogue state tracking (Henderson et al., 2014)), c","cbCaiqPnV7py9dy7","https://ap.wps.com/l/cbCaiqPnV7py9dy7","pdf",261681,1,11,"English","en",105,"# Abstract\n# Introduction","[{\"question\":\"What two kinds of consistency are required in task-oriented dialogues?\",\"answer\":\"They must be consistent internally across dialogue turns and consistent with the conversational domain knowledge. Internal consistency ensures coherence across the participants, while external consistency ensures correct reflection of domain facts.\"},{\"question\":\"How does the paper model dialogue consistency?\",\"answer\":\"It formulates dialogue consistency as a Constraint Satisfaction Problem (CSP). Dialogue-referencing segments are represented as variables, while linguistic, conversational, and domain-based properties are represented as constraints.\"},{\"question\":\"What did the experiments show about LLM-based re-lexicalization?\",\"answer\":\"The CSP solver effectively detects inconsistencies, while consistent dialogue re-lexicalization remains difficult for state-of-the-art LLMs, with only 0.15 accuracy versus the CSP solver. The ablation study indicates that constraints derived from domain knowledge are hardest for the models to satisfy.\"}]",1784173818,28,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":27},"evaluating-task-oriented-dialogue-consistency-through-constraint-satisfaction","",{"@graph":35,"@context":85},[36,53,68],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/evaluating-task-oriented-dialogue-consistency-through-constraint-satisfaction/81497/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-17","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What two kinds of consistency are required in task-oriented dialogues?","Question",{"text":75,"@type":76},"They must be consistent internally across dialogue turns and consistent with the conversational domain knowledge. Internal consistency ensures coherence across the participants, while external consistency ensures correct reflection of domain facts.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"How does the paper model dialogue consistency?",{"text":80,"@type":76},"It formulates dialogue consistency as a Constraint Satisfaction Problem (CSP). Dialogue-referencing segments are represented as variables, while linguistic, conversational, and domain-based properties are represented as constraints.",{"name":82,"@type":73,"acceptedAnswer":83},"What did the experiments show about LLM-based re-lexicalization?",{"text":84,"@type":76},"The CSP solver effectively detects inconsistencies, while consistent dialogue re-lexicalization remains difficult for state-of-the-art LLMs, with only 0.15 accuracy versus the CSP solver. The ablation study indicates that constraints derived from domain knowledge are hardest for the models to satisfy.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":45,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":45,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":45,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":45,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":45,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":45,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":45,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]