[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-85258-en":3,"doc-seo-85258-105":29,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},85258,1374391974564,"Clementine","https://ap-avatar.wpscdn.com/avatar/14000253aa45c000a9e?x-image-process=image/resize,m_fixed,w_180,h_180&k=1779874745381141002",8,"Research & Report","Capabilities of Claude Fable 5 on Biomedical Challenge Problems","Frontier language models are increasingly tested on biomedical benchmarks, yet many evaluations are weakened by two issues: legacy benchmarks are near-saturated, and open-ended answers are often judged by other language models. This study evaluates Claude Fable 5, Anthropic’s most capable publicly available model, across eight biomedical benchmarks (four text, four multimodal) with deterministic scoring against fixed answer keys. Refusal is tracked separately, revealing benchmark-dependent refusal rates that alter measured accuracy; after excluding refused items, Fable 5 matches or exceeds baselines.","arXiv :2607 . 10849v 1 [ cs .CL] 12 Jul 2026  \nCapabilities of Claude Fable 5 on Biomedical Challenge Problems  \nDominic Okonkwo 1* , Magnus Hodgson 1 , Temitope I. David2 , Susan Adanna Ihejirika3  \n1 School of Computing, University of Georgia  \n2Department of Chemistry, University of Illinois  \n3Institute of Bioinformatics, University of Georgia  \nAbstract  \nFrontier language models are increasingly evaluated on biomedical benchmarks, but two problems undermine most published evaluations: legacy benchmarks are near-saturated, and open-ended responses are graded by other language models. We evaluate Claude Fable 5, Anthropic’s most capable publicly available model, across eight biomedical benchmarks, four text and four multimodal, using deterministic scoring against fixed answer keys throughout. We include two Claude predecessors and GPT-5 as baselines. Refusal is tracked as a distinct outcome in every result table. That decision produces the paper’s central finding. Fable 5 refuses between 8.0% and 99.4% of questions depending on the benchmark, a pattern absent in both predecessors and in GPT-5 . Once refused items are excluded from the denominator, Fable 5’s accuracy exceeds or meets every other model on every benchmark in this study. We identify two distinguishable refusal patterns: one concentrating in basic-science and mechanism content across MedQA and MedXpertQA MM, confirmed independently on two benchmarks using each benchmark’s own category labels; and a separate disease-domain pattern on RareBench, where inborn metabolic disease presentations are refused near-universally while adult-onset autoimmune presentations are not. The primary constraint on Fable 5’s biomedical usefulness is willingness to engage, not capability once it does.  \n100  \n75  \n50  \n25  \n0  \nMedQA  \n\n| 95.0 94.3 96.0 |  |  |\n| --- | --- | --- |\n| 17% refused |  |  |\n| 79.7 |  |  |\n|  |  |  |\n|  |  |  |\n|  |  |  |\n\nFable 5 Opus 4.8 Opus 4.6 GPT-5  \n100  \n75  \n50  \n25  \n0  \nPubMedQA  \n\n|  |  |  |\n| --- | --- | --- |\n| 20% refused 75.2 76.6 71.7 |  |  |\n| 65.0 |  |  |\n|  |  |  |\n|  |  |  |\n|  |  |  |\n\nFable 5 Opus 4.8 Opus 4.6 GPT-5  \n100  \n75  \n50  \n25  \n0  \nMedXpertQA (Text)  \n\n|  |  |  |  |\n| --- | --- | --- | --- |\n|  |  |  |  |\n| 12% refused |  |  |  |\n| 57.7\u003Cbr>52.7 50.5 |  |  | 54.7 |\n|  |  |  |  |\n|  |  |  |  |\n\nFable 5 Opus 4.8 Opus 4.6 GPT-5  \nRareBench (R@10)  \n100  \n75  \n50  \n25  \n0  \n\n|  |\n| --- |\n|  |\n| 56.8\u003Cbr>47.5 49.4 |\n|  |\n| 99% refused\u003Cbr>0.4 |\n\nFable 5 Opus 4.8 Opus 4.6 GPT-5  \nText Benchmarks   \n100  \n75  \n50  \n25  \n0  \nVQA-RAD  \n\n|  |  |  |\n| --- | --- | --- |\n|  |  |  |\n| 45.8 |  |  |\n| 40.2 | 35.2 | 33.5 |\n|  |  |  |\n\nFable 5 Opus 4.8 Opus 4.6 GPT-5  \n100  \n75  \n50  \n25  \n0  \nPathVQA  \n\n|  |  |  |  |  |\n| --- | --- | --- | --- | --- |\n|  |  |  |  |  |\n|  |  |  |  |  |\n| 42% refused 21.6 |  |  |  |  |\n| 17.0 |  |  | 8.8\u003Cbr> | 5.3\u003Cbr> |\n|  |  |  |  |  |\n\nFable 5 Opus 4.8 Opus 4.6 GPT-5  \n100  \n75  \n50  \n25  \n0  \nSLAKE  \n\n|  |  |  |  |  |\n| --- | --- | --- | --- | --- |\n|  |  |  |  |  |\n| 49.2 | 7% refused |  |  |  |\n|  |  | 43.6 | 38.0 | 40.4 |\n|  |  |  |  |  |\n|  |  |  |  |  |\n\nFable 5 Opus 4.8 Opus 4.6 GPT-5  \nMedXpertQA MM  \n100  \n75  \n50  \n25  \n0  \n\n|  |  |  |\n| --- | --- | --- |\n| 8% refused\u003Cbr>73.8 |  |  |\n|  |  | 67.2\u003Cbr>62.3\u003Cbr>55.0 |\n|  |  |  |\n|  |  |  |\n\nFable 5 Opus 4.8 Opus 4.6 GPT-5  \nMultimodal Benchmarks   \nOverview of the eight biomedical benchmarks evaluated in this study. Raw accuracy across eight biomedical benchmarks; refusal rate annotated in red where non-negligible (≥5%) .  \n1. Introduction  \nBiomedical evaluation of language models has not kept pace with the models themselves. Every new frontier model claims stronger scientific reasoning, and every new model report leads with a wall of benchmark scores. But two specific practices have quietly undermined what those scores mean. First, models have gotten good enough that many established exams no longer separate one system from the next, with frontier mode","cbCaiaxUz045nX6q","https://ap.wps.com/l/cbCaiaxUz045nX6q","pdf",691121,1,15,"English","en",105,"# Introduction\n## Evaluation setup and scoring principles\n# Results on biomedical benchmarks\n## Text benchmarks\n## Multimodal benchmarks\n# Refusal patterns and analysis\n## Basic-science vs disease-domain refusal","[{\"question\":\"What two issues undermine many existing biomedical language-model evaluations?\",\"answer\":\"Legacy benchmarks are often near-saturated, and grading for open-ended responses is frequently outsourced to other language models, making the evaluation dependent on the judge’s reliability.\"},{\"question\":\"How is Claude Fable 5 evaluated in this study?\",\"answer\":\"The model is tested across eight biomedical benchmarks (four text, four multimodal) using deterministic scoring against fixed answer keys in a single-shot setting.\"},{\"question\":\"Why does tracking refusal matter for the reported accuracy?\",\"answer\":\"Fable 5 refuses between 8.0% and 99.4% of questions depending on the benchmark. Excluding refused items from the accuracy denominator changes the outcome, after which Fable 5 meets or exceeds other models on every benchmark.\"}]",1784202122,38,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":27},"capabilities-of-claude-fable-5-on-biomedical-challenge-problems","",{"@graph":35,"@context":85},[36,53,68],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/capabilities-of-claude-fable-5-on-biomedical-challenge-problems/85258/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-17","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What two issues undermine many existing biomedical language-model evaluations?","Question",{"text":75,"@type":76},"Legacy benchmarks are often near-saturated, and grading for open-ended responses is frequently outsourced to other language models, making the evaluation dependent on the judge’s reliability.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"How is Claude Fable 5 evaluated in this study?",{"text":80,"@type":76},"The model is tested across eight biomedical benchmarks (four text, four multimodal) using deterministic scoring against fixed answer keys in a single-shot setting.",{"name":82,"@type":73,"acceptedAnswer":83},"Why does tracking refusal matter for the reported accuracy?",{"text":84,"@type":76},"Fable 5 refuses between 8.0% and 99.4% of questions depending on the benchmark. Excluding refused items from the accuracy denominator changes the outcome, after which Fable 5 meets or exceeds other models on every benchmark.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":45,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":45,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":45,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":45,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":45,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":45,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":45,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]