[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-seo-320747-105":3,"detail-sidebar-cat-0-en-105":81,"doc-detail-320747-en":131},{"code":4,"msg":5,"data":6},0,"ok",{"site_id":7,"language":8,"slug":9,"title":10,"keywords":11,"description":12,"schema_data":13,"social_meta":74,"head_meta":76,"extra_data":78,"updated_unix":80},105,"en","re-evaluating-gpt-4s-bar-exam-performance-eric-martinez","Re-Evaluating GPT-4’s Bar Exam Performance - Eric Martínez","","The paper evaluates reported claims that GPT-4 achieved a 90th-percentile performance on the Uniform Bar Exam (UBE). It examines methodological challenges in documenting and verifying the percentile statement, including skew from repeat test-takers and comparisons across administrations. Results also challenge the reported scaled UBE score and show grading issues for essay-related components. It further studies hyperparameters, finding no temperature effect but a benefit from few-shot chain-of-thought prompting.",{"@graph":14,"@context":73},[15,34,56],{"@type":16,"itemListElement":17},"BreadcrumbList",[18,23,27,31],{"item":19,"name":20,"@type":21,"position":22},"https://docshare.wps.com","Home","ListItem",1,{"item":24,"name":25,"@type":21,"position":26},"https://docshare.wps.com/document/","Document",2,{"item":28,"name":29,"@type":21,"position":30},"https://docshare.wps.com/document/research-report/","Research & Report",3,{"item":32,"name":10,"@type":21,"position":33},"https://docshare.wps.com/document/re-evaluating-gpt-4s-bar-exam-performance-eric-martinez/320747/",4,{"url":32,"name":10,"@type":35,"image":36,"author":41,"headline":10,"publisher":44,"fileFormat":47,"inLanguage":8,"description":12,"dateModified":48,"datePublished":49,"encodingFormat":47,"isAccessibleForFree":50,"interactionStatistic":51},"DigitalDocument",{"url":37,"@type":38,"width":39,"height":40},"https://docshare.wps.com/thumbnails/re-evaluating-gpt-4s-bar-exam-performance-eric-martinez/320747.png","ImageObject",300,407,{"name":42,"@type":43},"Maeve","Person",{"url":19,"name":45,"@type":46},"DocShare","Organization","application/pdf","2026-11-08","2026-09-21",true,{"@type":52,"interactionType":53,"userInteractionCount":55},"InteractionCounter",{"@type":54},"ViewAction",18,{"@type":57,"mainEntity":58},"FAQPage",[59,65,69],{"name":60,"@type":61,"acceptedAnswer":62},"What does the paper question about GPT-4’s Bar Exam results?","Question",{"text":63,"@type":64},"It questions the validity of the widely publicized 90th-percentile claim on the Uniform Bar Exam and also evaluates the reported scaled UBE score.","Answer",{"name":66,"@type":61,"acceptedAnswer":67},"How do the findings estimate GPT-4’s UBE percentile under different groups and assumptions?",{"text":68,"@type":64},"Estimates vary across analyses: overall results are reported as overinflated compared with the general test-taking population, with lower percentiles when focusing on first-time test takers and further drops when restricting to those who passed the exam.",{"name":70,"@type":61,"acceptedAnswer":71},"Do hyperparameter changes affect GPT-4’s MBE performance?",{"text":72,"@type":64},"Adjusting temperature settings shows no significant effect, while few-shot chain-of-thought prompting has a significant positive effect compared with basic zero-shot prompting.","https://schema.org",{"og:url":32,"og:type":75,"og:title":10,"og:site_name":45,"og:description":12},"article",{"robots":77,"canonical":32},"index,follow",{"doc_id":79,"site_id":7},320747,1790164318,{"code":4,"msg":82,"data":83},"success",[84,88,92,96,101,106,111,115,120,123,127],{"id":22,"doc_module":4,"doc_module_name":25,"category_name":85,"show_sort_weight":86,"slug":87},"Story & Novel",90,"story-novel",{"id":26,"doc_module":4,"doc_module_name":25,"category_name":89,"show_sort_weight":90,"slug":91},"Literature",80,"literature",{"id":33,"doc_module":4,"doc_module_name":25,"category_name":93,"show_sort_weight":94,"slug":95},"Exam",70,"exam",{"id":97,"doc_module":4,"doc_module_name":25,"category_name":98,"show_sort_weight":99,"slug":100},5,"Comic",60,"comic",{"id":102,"doc_module":4,"doc_module_name":25,"category_name":103,"show_sort_weight":104,"slug":105},6,"Technology",50,"technology",{"id":107,"doc_module":4,"doc_module_name":25,"category_name":108,"show_sort_weight":109,"slug":110},7,"Healthcare",40,"healthcare",{"id":112,"doc_module":4,"doc_module_name":25,"category_name":29,"show_sort_weight":113,"slug":114},8,30,"research-report",{"id":116,"doc_module":4,"doc_module_name":25,"category_name":117,"show_sort_weight":118,"slug":119},9,"Religion & Spirituality",20,"religion-spirituality",{"id":118,"doc_module":4,"doc_module_name":25,"category_name":121,"show_sort_weight":118,"slug":122},"World Cup","world-cup",{"id":124,"doc_module":4,"doc_module_name":25,"category_name":125,"show_sort_weight":124,"slug":126},10,"Lifestyle","lifestyle",{"id":128,"doc_module":4,"doc_module_name":25,"category_name":129,"show_sort_weight":97,"slug":130},19,"General","general",{"code":4,"msg":82,"data":132},{"doc_id":79,"user_id":133,"nickname":42,"user_avatar":134,"doc_module":4,"category_id":112,"category_name":29,"doc_title":10,"doc_description":12,"doc_content":135,"file_id":136,"file_url":137,"file_type":138,"file_size":139,"view_count":55,"is_deleted":4,"is_public":22,"is_downloadable":22,"audit_status":22,"page_count":140,"language":141,"language_code":8,"site_id":7,"html_lang":8,"table_of_contents":142,"faqs":143,"seo_title":144,"seo_description":12,"update_tm":145,"read_time":146,"preview_page_count":4},5909877438554,"https://ap-avatar.wpscdn.com/avatar/5600025385ad2bf12a7?_k=1778553567797529272","Re-Evaluating GPT-4’s Bar Exam Performance  \nEric Martínez 1  \nAbstract  \nPerhaps the most widely touted of GPT-4’s at-launch, zero-shot capabilities has been its reported 90th-percentile performance on the Uniform Bar Exam. This paper begins by investigating the methodological challenges in documenting and verifying the 90th-percentile claim, presenting four sets of findings that indicate that OpenAI’s estimates of GPT-4’s UBE percentile are overinflated.  \nFirst, although GPT-4’s UBE score nears the 90th percentile when examining approximate conversions from February administrations of the Illinois Bar Exam, these estimates are heavily skewed towards repeat test-takers who failed the July administration and score significantly lower than the general test-taking population. Second, data from a recent July administration of the same exam suggests GPT-4’s overall UBE percentile was below the 69th percentile, and ∼48th percentile on essays. Third, examining official NCBE data and using several conservative statistical assumptions, GPT-4’s performance against firsttime test takers is estimated to be ∼62nd percentile, including ∼42nd percentile on essays. Fourth, when examining only those who passed the exam (i.e. licensed or license-pending attorneys), GPT-4’s performance is estimated to drop to ∼48th percentile overall, and ∼ 15th percentile on essays.  \nIn addition to investigating the validity of the percentile claim, the paper also investigates the validity of GPT-4’s reported scaled UBE score of 298 . The paper successfully replicates the MBE score, but highlights several methodological issues in the grading of the MPT + MEE components of the exam, which call into question the validity of the reported essay score.  \nFinally, the paper investigates the effect of different hyperparameter combinations on GPT-4’s MBE performance, finding no significant effect of adjusting temperature settings, and a significant effect of few-shot chain-of-thought prompting over basic zero-shot prompting.  \nTaken together, these findings carry timely insights for the desirability and feasibility of outsourcing legally relevant tasks to AI models, as well as for the  \n1 PhD Student in Cognitive Science, Massachusetts Institute of Technology (MIT); JD, Harvard Law School. Email: [ericmart@mit.edu](ericmart@mit.edu).  \nSpecial thanks to Aixiu An, Christoph Winter, Matthijs Maas, Markus Anderljung, John Bliss, Daniel Katz, Jonas Schuett, Michelle Hung, Ted Gibson, Jon Choi, Dan Schwarcz, and Rosemary Reshetar for comments, guidance and feedback.  \nAll code for this paper is publicly available at the following repository link: code  \nElectronic copy available at: [https://ssrn.com/abstract=4441311](https://ssrn.com/abstract=4441311)  \nimportance for AI developers to implement rigorous and transparent capabilities evaluations to help secure safe and trustworthy AI.  \n1. Introduction  \nOn March 14th, 2023, OpenAI launched GPT-4, said to be the latest milestone in the company’s effort in scaling up deep learning [1] . As part of its launch, OpenAI revealed details regarding the model’s “human-level performance on various professional and academic benchmarks” [1] . Perhaps none of these capabilities was as widely publicized as GPT-4’s performance on the Uniform Bar Examination, with OpenAI prominently displaying on various pages of its website and technical report that GPT-4 scored in or around the “90th percentile,”[1–3] or “the top 10% of test-takers,” [1, 2] and various prominent media outlets [4–8] and legal scholars [9] resharing and discussing the implications of these results for the legal profession and the future of AI.  \nOf course, assessing the capabilities of an AI system as compared to those of a human is no easy task [10–15], and in the context of the legal profession specifically, there are various reasons to doubt the usefulness of the bar exam as a proxy for lawyerly competence (both for humans and AI systems), given that, for example: (a","cbCaijxe5lrhLFz3","https://ap.wps.com/l/cbCaijxe5lrhLFz3","pdf",563142,25,"English","# Abstract\n# Introduction","[{\"question\":\"What does the paper question about GPT-4’s Bar Exam results?\",\"answer\":\"It questions the validity of the widely publicized 90th-percentile claim on the Uniform Bar Exam and also evaluates the reported scaled UBE score.\"},{\"question\":\"How do the findings estimate GPT-4’s UBE percentile under different groups and assumptions?\",\"answer\":\"Estimates vary across analyses: overall results are reported as overinflated compared with the general test-taking population, with lower percentiles when focusing on first-time test takers and further drops when restricting to those who passed the exam.\"},{\"question\":\"Do hyperparameter changes affect GPT-4’s MBE performance?\",\"answer\":\"Adjusting temperature settings shows no significant effect, while few-shot chain-of-thought prompting has a significant positive effect compared with basic zero-shot prompting.\"}]","Re-Evaluating GPT-4’s Bar Exam Performance - Eric Martínez | PDF",1789952429,63]