[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-seo-308803-105":3,"detail-sidebar-cat-0-en-105":81,"doc-detail-308803-en":131},{"code":4,"msg":5,"data":6},0,"ok",{"site_id":7,"language":8,"slug":9,"title":10,"keywords":11,"description":12,"schema_data":13,"social_meta":74,"head_meta":76,"extra_data":78,"updated_unix":80},105,"en","large-legal-fictions-profiling-legal-hallucinations-in-large-language-models-research-overview","Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models - Research overview","","Large Legal Fictions examines whether large language models (LLMs) know the law and quantifies “hallucinations” that produce outputs inconsistent with legal facts. The study presents systematic evidence from public-facing LLMs, tracking how hallucination rates vary across jurisdictions, courts, time periods, and cases. Using ChatGPT 4 and other models, the findings show hallucination at least 58% of the time, difficulties in self-prediction, and frequent acceptance of users’ incorrect legal assumptions.",{"@graph":14,"@context":73},[15,34,56],{"@type":16,"itemListElement":17},"BreadcrumbList",[18,23,27,31],{"item":19,"name":20,"@type":21,"position":22},"https://docshare.wps.com","Home","ListItem",1,{"item":24,"name":25,"@type":21,"position":26},"https://docshare.wps.com/document/","Document",2,{"item":28,"name":29,"@type":21,"position":30},"https://docshare.wps.com/document/research-report/","Research & Report",3,{"item":32,"name":10,"@type":21,"position":33},"https://docshare.wps.com/document/large-legal-fictions-profiling-legal-hallucinations-in-large-language-models-research-overview/308803/",4,{"url":32,"name":10,"@type":35,"image":36,"author":41,"headline":10,"publisher":44,"fileFormat":47,"inLanguage":8,"description":12,"dateModified":48,"datePublished":49,"encodingFormat":47,"isAccessibleForFree":50,"interactionStatistic":51},"DigitalDocument",{"url":37,"@type":38,"width":39,"height":40},"https://docshare.wps.com/thumbnails/large-legal-fictions-profiling-legal-hallucinations-in-large-language-models-research-overview/308803.png","ImageObject",300,407,{"name":42,"@type":43},"Valentina","Person",{"url":19,"name":45,"@type":46},"DocShare","Organization","application/pdf","2026-11-09","2026-09-20",true,{"@type":52,"interactionType":53,"userInteractionCount":55},"InteractionCounter",{"@type":54},"ViewAction",21,{"@type":57,"mainEntity":58},"FAQPage",[59,65,69],{"name":60,"@type":61,"acceptedAnswer":62},"What problem does the paper focus on?","Question",{"text":63,"@type":64},"The paper focuses on “hallucinations” in large language models—textual outputs that do not align with legal facts and doctrine.","Answer",{"name":66,"@type":61,"acceptedAnswer":67},"How does the paper assess hallucinations in public-facing LLMs?",{"text":68,"@type":64},"It documents hallucinations using public models such as ChatGPT 4 and analyzes patterns across jurisdictions, courts, time periods, and cases.",{"name":70,"@type":61,"acceptedAnswer":71},"What do the results indicate about hallucination frequency and user trust?",{"text":72,"@type":64},"The study reports LLM hallucinations at least 58% of the time, challenges in predicting their own hallucinations, and frequent uncritical acceptance of users’ incorrect legal assumptions.","https://schema.org",{"og:url":32,"og:type":75,"og:title":10,"og:site_name":45,"og:description":12},"article",{"robots":77,"canonical":32},"index,follow",{"doc_id":79,"site_id":7},308803,1789872407,{"code":4,"msg":82,"data":83},"success",[84,88,92,96,101,106,111,115,120,123,127],{"id":22,"doc_module":4,"doc_module_name":25,"category_name":85,"show_sort_weight":86,"slug":87},"Story & Novel",90,"story-novel",{"id":26,"doc_module":4,"doc_module_name":25,"category_name":89,"show_sort_weight":90,"slug":91},"Literature",80,"literature",{"id":33,"doc_module":4,"doc_module_name":25,"category_name":93,"show_sort_weight":94,"slug":95},"Exam",70,"exam",{"id":97,"doc_module":4,"doc_module_name":25,"category_name":98,"show_sort_weight":99,"slug":100},5,"Comic",60,"comic",{"id":102,"doc_module":4,"doc_module_name":25,"category_name":103,"show_sort_weight":104,"slug":105},6,"Technology",50,"technology",{"id":107,"doc_module":4,"doc_module_name":25,"category_name":108,"show_sort_weight":109,"slug":110},7,"Healthcare",40,"healthcare",{"id":112,"doc_module":4,"doc_module_name":25,"category_name":29,"show_sort_weight":113,"slug":114},8,30,"research-report",{"id":116,"doc_module":4,"doc_module_name":25,"category_name":117,"show_sort_weight":118,"slug":119},9,"Religion & Spirituality",20,"religion-spirituality",{"id":118,"doc_module":4,"doc_module_name":25,"category_name":121,"show_sort_weight":118,"slug":122},"World Cup","world-cup",{"id":124,"doc_module":4,"doc_module_name":25,"category_name":125,"show_sort_weight":124,"slug":126},10,"Lifestyle","lifestyle",{"id":128,"doc_module":4,"doc_module_name":25,"category_name":129,"show_sort_weight":97,"slug":130},19,"General","general",{"code":4,"msg":82,"data":132},{"doc_id":79,"user_id":133,"nickname":42,"user_avatar":134,"doc_module":4,"category_id":112,"category_name":29,"doc_title":10,"doc_description":12,"doc_content":135,"file_id":136,"file_url":137,"file_type":138,"file_size":139,"view_count":55,"is_deleted":4,"is_public":22,"is_downloadable":22,"audit_status":22,"page_count":113,"language":140,"language_code":8,"site_id":7,"html_lang":8,"table_of_contents":141,"faqs":142,"seo_title":143,"seo_description":12,"update_tm":80,"read_time":144,"preview_page_count":4},13056703020460,"https://ap-avatar.wpscdn.com/avatar/be000253dac470eee5d?_k=1778207105932848923","Journal of Legal Analysis, 2024, 16, 64–93 [https://doi.org/10.1093/jla/laae003](https://doi.org/10.1093/jla/laae003)[ ](https://doi.org/10.1093/jla/laae003)Advance access publication 26 June 2024 Article  \nLarge Legal Fictions: Profiling Legal Hallucinations in Large Language Models  \nMatthew Dahl* , Varun Magesh†, Mirac Suzgun‡, and Daniel E. Ho§  \nAbstract  \nDo large language models (LLMs) know the law? LLMs are increasingly being used to augment legal practice, education, and research, yet their revolutionary potential is threatened by the presence of“hallucinations”—textual output that is not consistent with legal facts. We present the first systematic evidence of these hallucinations in public-facing LLMs, documenting trends across jurisdictions, courts, time periods, and cases. Using OpenAI’s ChatGPT 4 and other public models, we show that LLMs hallucinate at least 58% of the time, struggle to predict their own hallucinations, and often uncritically accept users’ incorrect legal assumptions. We conclude by cautioning against the rapid and unsupervised integration of popular LLMs into legal tasks, and we develop a typology of legal hallucinations to guide future research in this area.  \n1 INTRODUCTION  \nHow well do large language models (LLMs) know American case law? Modern LLMs such as OpenAI’s ChatGPT—tools trained on vast amounts of textual data to predict the next token in a sequence—are driving a transformation in the legal world, from legal education (Choi and Schwarcz 2024), to legal research (Livermore, Herron, and Rockmore 2024), to legal practice itself (Rodgers, Armour, and Sako 2023) . Indeed, recent versions of these artificial intelligence (AI) models seem to excel at law-related tasks, such as first-year law school exams (Choi et al. 2022), the uniform bar exam (Katz et al. 2023), statutory reasoning (Blair-Stanek, Holzenberger, and Van Durme 2023), and issue-rule-applicationconclusion (IRAC) analysis (Guha et al. 2023) . But despite the revolutionary potential of these models, a key challenge remains: the issue of“hallucinations.” LLMs are liable to generate language that is inconsistent with current legal doctrine and case law, and, in the legal field, where adherence to authorities is paramount, unfaithful or imprecise interpretations of the law can lead to nonsensical—or worse, harmful and inaccurate—legal advice or decisions.  \nIn this work, we present the first evidence documenting the nature, frequency, and correlates of these hallucinations. In doing so, we shed systematic, empirical light on a phenomenon that has so far only received anecdotal treatment in the literature. For example, much media attention has been directed toward a Manhattan lawyer who faced sanctions for using ChatGPT to generate fictional  \n* J.D. Candidate, Yale Law School, and Ph.D. Student, Yale University Department of Political Science, USA. E-mail: matthew. [dahl@yale.edu](dahl@yale.edu).  \n† Research Fellow, Regulation, Evaluation, and Governance Lab, Stanford University, USA.  \n‡ J.D. Candidate, Stanford Law School, and Ph.D. Candidate, Stanford University Department of Computer Science, USA.  \n§ William Benjamin Scott and Luna M. Scott Professor of Law, Professor of Political Science, Senior Fellow at Stanford Institute for Economic Policy Research, and Faculty Director of the Regulation, Evaluation, and Governance Lab, Stanford University, USA. We thank Neel Guha, Sandy Handan-Nader, Adam T. Kalai, Peter Maldonado, Chris Manning, Joel Niklaus, Kit Rodolfa, Faiz Surani, Andrea Vallebueno, and three anonymous reviewers for helpful discussions and feedback. Our replication materials are available in the Harvard Dataverse ([https://doi.org/10.7910/DVN/V4ON8H](https://doi.org/10.7910/DVN/V4ON8H)) and our data for future benchmarking is available on HuggingFace ([https://huggingface.co/datasets/reglab/legal_hallucinations](https://huggingface.co/datasets/reglab/legal_hallucinations)) .  \n© The Author(s) 2024. Published by Ox","cbCailFGKpWm1fza","https://ap.wps.com/l/cbCailFGKpWm1fza","pdf",3676867,"English","# 1 Introduction\n## Research motivation and problem framing\n## Evidence approach and key findings","[{\"question\":\"What problem does the paper focus on?\",\"answer\":\"The paper focuses on “hallucinations” in large language models—textual outputs that do not align with legal facts and doctrine.\"},{\"question\":\"How does the paper assess hallucinations in public-facing LLMs?\",\"answer\":\"It documents hallucinations using public models such as ChatGPT 4 and analyzes patterns across jurisdictions, courts, time periods, and cases.\"},{\"question\":\"What do the results indicate about hallucination frequency and user trust?\",\"answer\":\"The study reports LLM hallucinations at least 58% of the time, challenges in predicting their own hallucinations, and frequent uncritical acceptance of users’ incorrect legal assumptions.\"}]","Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models - Research overview | PDF",76]