[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-81713-en":3,"doc-seo-81713-105":29,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},81713,3848291630094,"Emma Wilson","https://eur-avatar.wpscdn.com/davatar_085a072bc5b1113ac321206ff7593b45",8,"Research & Report","LLMs in the Real World: Evaluating “AI” in Emergency Contexts","LLMs in the Real World: Evaluating “AI” in Emergency Contexts is a research paper that calls for stronger public communication of AI findings and clearer accountability for real-world deployments. The work studies the early rollout of an LLM-based text-to-911 machine translation system advertised as supporting 55 languages for emergency use where direct operator contact may be difficult. It identifies common misconceptions, argues that overlooked “easy” problems can drive failures, and offers concrete recommendations across development and deployment stages for stakeholders.","LLMs in the Real World: Evaluating “AI” in Emergency Contexts  \nSara Court  \nThe Ohio State University [court.22@osu.edu](court.22@osu.edu)  \nLara Downing  \nCommunity Refugee & Immigration Services (CRIS) [ldowning@cris-ohio.org](ldowning@cris-ohio.org)  \nMicha Elsner  \nThe Ohio State University [elsner.14@osu.edu](elsner.14@osu.edu)  \narXiv :2607 .00019v1 [ cs .CY] 29 May 2026  \nAbstract  \nThis paper offers a call to action. We urge our colleagues in the research community to playa greater role in the articulation of our findings to the public. To illustrate the stakes we present a case study on the initial stages of an LLM-based machine translation application’s deployment in a real-world context: a text-2- 911 system advertising capabilities in 55 languages for use in emergencies in which it maybe difficult to call operators directly. We identify a number of common misconceptions about technologies such as these, concluding with a set of concrete recommendations and best practices for stakeholders at every stage of the development and deployment pipeline. While the advancement of scientific research often lies in solving the “hard” problems, we argue it is often the “easy” ones—problems for which the latest technology is often unnecessary—that are most overlooked.  \n1 Introduction  \nDespite considerable overlap between academic and industry-based developers of Large Language Models (LLMs) and related technologies (Abdalla et al., 2023), it seems Natural Language Processing (NLP) researchers have a science outreach problem. As our research findings continue to drive the development of some of the most quickly adopted user-facing applications to date (De Brugger, 2023), the findings themselves—and their realworld implications—are too often lost in the hype. Artificial Intelligence (AI) is increasingly being offered as an inevitable solution to some of humanity’s largest problems (Eubanks, 2018 ; Byrum and Benjamin, 2022 ; Benjamin, 2024 ; Center for Democracy & Technology, 2025b) .  \nAs the outputs of modern NLP research are taken up and applied in commercial products, an information gap has developed between those designing NLP applications and their end users. For example,  \nresearchers may take it for granted that a model performs worse with lower-resourced languages (Silva et al., 2024), or that its performance degrades in response to inputs from outside its training domain (Wu et al., 2024 ; Li et al., 2025) . However, decision makers deploying NLP products in essential services like law enforcement and emergency response may not be aware of these limitations. Many may find themselves under pressure to find ways to acquire and integrate AI tools, but are left to navigate the AI software market without the knowledge needed to properly evaluate them, mitigate their risks, and ensure that they’re as safe, ethical, and effective as possible.  \nThis failure to effectively communicate our research findings to the public— including to those developing and selling consumer-facing applications— is not unique to NLP. Cryptographers have developed easy-to-access, publicly available code for secure public-key communication, but surveys of these systems in actual use persistently reveal that end users continue to create security flaws by misusing the APIs (Choudhari et al., 2021 ; Lazar et al., 2014) . The NLP research community now faces a similar problem. Although basic tools for transparency and evaluation like model cards (Mitchell et al., 2019) have existed for years, they still aren’t widely used in commercial settings.  \nMost consumers of NLP technologies have little other than promotional materials to go off of as they navigate the complex landscape of tools marketed to the public as “AI.” The result, we claim, is both an information imbalance and an accountability gap: NLP products are increasingly being sold to consumers in both public and private sectors, including in high stakes contexts such as policing (United States v.","cbCaikydqes7GznN","https://ap.wps.com/l/cbCaikydqes7GznN","pdf",275484,1,13,"English","en",105,"# Abstract\n# Introduction\n# Case Study: Text-2-911 Service\n## Language Access and Technology in Emergencies","[{\"question\":\"What is the main purpose of the paper?\",\"answer\":\"The paper calls for a stronger role for the research community in communicating findings to the public and improving how AI deployments are understood and evaluated in high-stakes settings.\"},{\"question\":\"What case study does the paper analyze?\",\"answer\":\"It analyzes the initial deployment stages of an LLM-based text-to-911 machine translation application used in emergency contexts at a U.S. 911 center.\"},{\"question\":\"What does the paper conclude about common risks and misconceptions?\",\"answer\":\"It identifies common misconceptions about LLM-based technologies and argues that gaps in accountability can lead to risky and potentially harmful deployment, especially for vulnerable populations.\"}]",1784175570,33,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":27},"llms-in-the-real-world-evaluating-ai-in-emergency-contexts","",{"@graph":35,"@context":85},[36,53,68],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/llms-in-the-real-world-evaluating-ai-in-emergency-contexts/81713/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-20","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What is the main purpose of the paper?","Question",{"text":75,"@type":76},"The paper calls for a stronger role for the research community in communicating findings to the public and improving how AI deployments are understood and evaluated in high-stakes settings.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"What case study does the paper analyze?",{"text":80,"@type":76},"It analyzes the initial deployment stages of an LLM-based text-to-911 machine translation application used in emergency contexts at a U.S. 911 center.",{"name":82,"@type":73,"acceptedAnswer":83},"What does the paper conclude about common risks and misconceptions?",{"text":84,"@type":76},"It identifies common misconceptions about LLM-based technologies and argues that gaps in accountability can lead to risky and potentially harmful deployment, especially for vulnerable populations.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":45,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":45,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":45,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":45,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":45,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":45,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":45,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]