[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"detail-sidebar-cat-1-en-105":3,"doc-seo-572965-105":53,"doc-detail-572965-en":120},{"code":4,"msg":5,"data":6},0,"success",[7,14,19,24,29,34,39,44,49],{"id":8,"doc_module":9,"doc_module_name":10,"category_name":11,"show_sort_weight":12,"slug":13},11,1,"Template","Presentations",90,"presentations",{"id":15,"doc_module":9,"doc_module_name":10,"category_name":16,"show_sort_weight":17,"slug":18},12,"Resumes",80,"resumes",{"id":20,"doc_module":9,"doc_module_name":10,"category_name":21,"show_sort_weight":22,"slug":23},14,"Invoices",70,"invoices",{"id":25,"doc_module":9,"doc_module_name":10,"category_name":26,"show_sort_weight":27,"slug":28},15,"Posters",60,"posters",{"id":30,"doc_module":9,"doc_module_name":10,"category_name":31,"show_sort_weight":32,"slug":33},16,"Social Media",50,"social-media",{"id":35,"doc_module":9,"doc_module_name":10,"category_name":36,"show_sort_weight":37,"slug":38},17,"Forms",40,"forms",{"id":40,"doc_module":9,"doc_module_name":10,"category_name":41,"show_sort_weight":42,"slug":43},18,"Letters",30,"letters",{"id":45,"doc_module":9,"doc_module_name":10,"category_name":46,"show_sort_weight":47,"slug":48},21,"Paper Templates",5,"papers-templates",{"id":50,"doc_module":9,"doc_module_name":10,"category_name":51,"show_sort_weight":4,"slug":52},158,"General","general-158",{"code":4,"msg":54,"data":55},"ok",{"site_id":56,"language":57,"slug":58,"title":59,"keywords":60,"description":61,"schema_data":62,"social_meta":113,"head_meta":115,"extra_data":117,"updated_unix":119},105,"en","hashmarks-privacy-preserving-benchmarks-for-high-stakes-ai-evaluation","Hashmarks: Privacy-Preserving Benchmarks for High-Stakes AI Evaluation","","Growing demand exists for evaluating language model capabilities on sensitive subjects such as bioterrorism or cyberwarfare, yet traditional open benchmarks are unsuitable because they publish correct answers in human-readable form, while strict closed evaluations can hinder progress and reduce trust. The paper proposes hashmarking, an open evaluation protocol that cryptographically hashes reference solutions before publication. It then analyzes robustness against conventional attacks like rainbow table methods and against failure modes specific to advanced generative models.",{"@graph":63,"@context":112},[64,80,95],{"@type":65,"itemListElement":66},"BreadcrumbList",[67,71,74,77],{"item":68,"name":69,"@type":70,"position":9},"https://docshare.wps.com","Home","ListItem",{"item":72,"name":10,"@type":70,"position":73},"https://docshare.wps.com/template/",2,{"item":75,"name":51,"@type":70,"position":76},"https://docshare.wps.com/template/general/",3,{"item":78,"name":59,"@type":70,"position":79},"https://docshare.wps.com/template/hashmarks-privacy-preserving-benchmarks-for-high-stakes-ai-evaluation/572965/",4,{"url":78,"name":59,"@type":81,"author":82,"headline":59,"publisher":85,"fileFormat":88,"inLanguage":57,"description":61,"dateModified":89,"datePublished":89,"encodingFormat":88,"isAccessibleForFree":90,"interactionStatistic":91},"DigitalDocument",{"name":83,"@type":84},"LangkahRina","Person",{"url":68,"name":86,"@type":87},"DocShare","Organization","application/pdf","2026-10-13",true,{"@type":92,"interactionType":93,"userInteractionCount":4},"InteractionCounter",{"@type":94},"ViewAction",{"@type":96,"mainEntity":97},"FAQPage",[98,104,108],{"name":99,"@type":100,"acceptedAnswer":101},"Why are traditional open QA benchmarks not suitable for sensitive topics?","Question",{"text":102,"@type":103},"They require publishing reference solutions in human-readable form, which can effectively disclose actionable knowledge related to hazardous domains.","Answer",{"name":105,"@type":100,"acceptedAnswer":106},"What is hashmarking in this paper?",{"text":107,"@type":103},"Hashmarking is a protocol for evaluating language models in the open by cryptographically hashing reference solutions before publication, so correct answers are not disclosed directly.",{"name":109,"@type":100,"acceptedAnswer":110},"How does the paper evaluate security and resilience?",{"text":111,"@type":103},"It studies resilience against traditional cryptographic attack vectors such as rainbow table attacks, and also against failure modes that are unique to increasingly capable generative models.","https://schema.org",{"og:url":78,"og:type":114,"og:title":59,"og:site_name":86,"og:description":61},"article",{"robots":116,"canonical":78},"index,follow",{"doc_id":118,"site_id":56},572965,1791930276,{"code":4,"msg":5,"data":121},{"doc_id":118,"user_id":122,"nickname":83,"user_avatar":123,"doc_module":9,"category_id":50,"category_name":51,"doc_title":59,"doc_description":61,"doc_content":124,"file_id":125,"file_url":126,"file_type":127,"file_size":128,"view_count":4,"is_deleted":4,"is_public":9,"is_downloadable":9,"audit_status":9,"page_count":30,"language":129,"language_code":57,"site_id":56,"html_lang":57,"table_of_contents":130,"faqs":131,"seo_title":132,"seo_description":61,"update_tm":119,"read_time":133},962090893776,"https://ap-avatar.wpscdn.com/davatar_155a257f0dc6eb9ab79c44ca47cae57d","arXiv :2312 .00645v2 [ cs .LG] 25 Dec 2023  \nHashmarks: Privacy-Preserving Benchmarks for High-Stakes AI Evaluation  \nPaul Bricman  \nStraumli AI  \n[paul@straumli. ai](paul@straumli. ai)  \nDecember 1, 2023  \nAbstract  \nThere is a growing need to gain insight into language model capabilities that relate to sensitive topics, such as bioterrorism or cyberwarfare. However, traditional open source benchmarks are not fit for the task, due to the associated practice of publishing the correct answers in humanreadable form. At the same time, enforcing mandatory closed-quarters evaluations might stifle development and erode trust. In this context, we propose hashmarking, a protocol for evaluating language models in the open without having to disclose the correct answers. In its simplest form, a hashmark is a benchmark whose reference solutions have been cryptographically hashed prior to publication. Following an overview of the proposed evaluation protocol, we go on to assess its resilience against traditional attack vectors (e.g. rainbow table attacks), as well as against failure modes unique to increasingly capable generative models.  \n1 Introduction  \n1.1 Background & Motivation  \nTraditional question-answering (QA) benchmarks have played a crucial role in shaping the trajectory of AI development. They provide standardized metrics that facilitate fair comparisons across research groups and measure progress within the field in a reproducible way [1] . For instance, capabilities related to mathematics can generally be gauged by assessing model performance in producing or selecting correct answers for exam questions related to mathematics [2, 3] . Similar \"AI exams\" have been used to assess model performance on topics ranging from STEM [4, 5 , 6 , 7] to humanities [4, 8] . In fact, prior work has highlighted the possibility of framing the vast majority of established natural language processing tasks as question-answering tasks [9] .  \nTraditional QA benchmarks are typically sourced from crowd-workers, members of the group developing the benchmark, or a mix of the two [1, 10] . They  \ntypically contain a large number of data points representing individual exercises. Each data point, in turn, contains one question and one or several correct answers. Occasionally, a data point may also contain some relevant context in the form of a separate string, although the context can always be functionally merged with the question without loss of generality [11, 12] . In addition, when structured asa multiple-choice question, a data point also contains several distractor answers [4, 8] . These data points are then aggregated into one or more standardized files, and are finally made public for other groups to easily make use of them in evaluating language models [1, 13] .  \nWhile traditional QA benchmarks have been instrumental in assessing models across a wide range of benign domains, there is a growing need to better understand capabilities that relate to sensitive topics, such as bioterrorism or cyberwarfare [14] . Greater insight into, for instance, AI-enabled biorisk, could help calibrate the stringency of regulations and policies in a way that is proportional to the associated threat. Underestimating the level of risk could, among others, enable bad actors to more easily weaponize this technology asa means of causing harm at scale. Conversely, overestimating the level of risk posed by a given class of generative models could also reduce the upsides being captured through beneficial usecases.  \nNaively applying traditional QA benchmarks on such sensitive topics, however, would face a major obstacle. By disclosing the reference solutions of the exam questions included in a hypothetical traditional benchmark on bioterrorism, one would inadvertently publish a veritable compendium on a topic better left undiscussed. The collection of questions coupled with correct answers would, in essence, provide a publicly-available \"FAQ\" on knowledge related to t","cbCairIkGkZVASl1","https://ap.wps.com/l/cbCairIkGkZVASl1","pdf",471563,"English","# Abstract\n# 1 Introduction\n## 1.1 Background & Motivation\n## 1.2 Related Work","[{\"question\":\"Why are traditional open QA benchmarks not suitable for sensitive topics?\",\"answer\":\"They require publishing reference solutions in human-readable form, which can effectively disclose actionable knowledge related to hazardous domains.\"},{\"question\":\"What is hashmarking in this paper?\",\"answer\":\"Hashmarking is a protocol for evaluating language models in the open by cryptographically hashing reference solutions before publication, so correct answers are not disclosed directly.\"},{\"question\":\"How does the paper evaluate security and resilience?\",\"answer\":\"It studies resilience against traditional cryptographic attack vectors such as rainbow table attacks, and also against failure modes that are unique to increasingly capable generative models.\"}]","Hashmarks: Privacy-Preserving Benchmarks for High-Stakes AI Evaluation | PDF",6]