[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-85713-en":3,"doc-seo-85713-105":29,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},85713,4398048950312,"Violet","https://ap-avatar.wpscdn.com/avatar/400002538284de19e3c?_k=1778320343897328908",8,"Research & Report","Judges Matter More Than Papers in Post-publication Research Assessment","Research assessment relies on expert evaluations, but human judgement contains noise and disagreement. It remains unclear whether rating differences reflect true variation in research quality or evaluator differences. Using a large post-publication peer review database, research quality ratings are found to be driven more by differences between evaluators than by differences between the evaluated papers. Multilevel variance decomposition across 239,521 ratings shows judge effects dominate rating variability, while directional bias measures explain under 1%.","Judges matter more than papers in post-publication research assessment  \nRobert Ward$, Alex Jones§, Lutz Bornmann*  \n$Bangor University  \nSchool of Psychology Brigantia Building; Penrallt Road Bangor UK LL57 2AS.  \nEmail: [r.ward@bangor.ac.uk](r.ward@bangor.ac.uk)  \n§Swansea University  \nSchool of Psychology  \nInstitute of Life Sciences Building 2  \nSwansea UK SA2 8PP.  \nEmail: [alex.l.jones@swansea.ac.uk](alex.l.jones@swansea.ac.uk)  \n*Science Policy and Strategy Department Administrative Headquarters of the Max Planck Society Hofgartenstr. 8,  \n80539 Munich, Germany.  \n[E-Mail: bornmann@gv.mpg.de](E-Mail: bornmann@gv.mpg.de)  \n[Corresponding author](Corresponding author).  \nAbstract  \nResearch assessment relies on expert evaluations, yet human judgement is noisy, and it is unclear whether differences in assessment arise primarily from differences in genuine research quality or from unwanted differences between evaluators. While numerous studies highlight disagreement and biases in research assessment, they have not quantified judgerelated noise relative to variation in the evaluated works. Here we show, in a large postpublication peer review database, that research assessment is driven more by differences between evaluators than by difference in the evaluated research. We partition variance in  \n239,521 research quality ratings assigned by 12,649 judges to 193,128 papers from the H1 Connect post-publication peer review platform. Using multilevel models, we decomposed judge-related variation into differences in overall severity and differences in the weighting of scientific attributes. We found that judge-related effects accounted for substantially more variance in ratings than the evaluated papers. In our most detailed model, judge-level effects and judge-specific slopes explained 61% of the total variance, whereas combined paper and journal-level effects accounted for only 7%. By contrast, examined measures of directional bias, such as author gender and global affiliation, explained less than 1% of the variance. We conclude that assessment outcomes were shaped more by the judges than by the papers themselves. Our results demonstrate the necessity of noise audits in high-stakes scientific evaluation.  \nKey words  \nResearch assessment, Peer review, Evaluator noise, Variance partitioning, Multilevel modelling, Systematic bias, Post-publication peer review  \nIntroduction  \nResearch assessment depends on expert judgement (Bornmann, 2011) . Decisions on publication, funding, hiring, and reputation are based on expert evaluations, meant to reflect underlying research quality. However, human judgement is inherently noisy, and disagreement among experts is found across domains, even when evaluation criteria are well defined (Allison et al, 2014; Bornmann et al., 2011; Cole et al, 1981; Dror & Cole, 2010; Pier et al, 2018; Simonsohn, 2007). Disagreement does not in itself imply system failure. For example, if research differs widely in quality but judges vary only slightly in their assessments, disagreement may have little overall effect on ranking outcomes. The central question is therefore not whether judges disagree, but whether variation in research assessment is driven primarily by differences in the quality of the evaluated works, or by differences between the experts doing the evaluations. The value of research assessment depends on this balance: Research assessment should be based on the quality of the research being evaluated, not on the personal preferences of the judges.  \nA useful framework for understanding variability in judgement distinguishes bias from noise. Bias refers to systematic deviations in evaluation, such as favouring authors, institutions, or research outcomes. Noise, in contrast, refers to variability in judgements that should in principle be the same. As emphasised by Kahneman et al. (2021), variability that appears as noise at one level of analysis may reflect systematic differences at another. For examp","cbCaitpztwMLo5Lo","https://ap.wps.com/l/cbCaitpztwMLo5Lo","pdf",372458,1,16,"English","en",105,"# Abstract\n# Introduction\n# Results","[{\"question\":\"What determines differences in research assessment outcomes according to the study?\",\"answer\":\"Differences between evaluators (judges) explain substantially more variance in ratings than differences between the evaluated papers.\"},{\"question\":\"How was the variance in research quality ratings analyzed?\",\"answer\":\"The study used multilevel models to partition rating variance into judge-related components and paper/journal-related components, including judge severity and attribute weighting.\"},{\"question\":\"Did the study find evidence of directional bias such as author gender or global affiliation?\",\"answer\":\"Directional bias measures like author gender and global affiliation explained less than 1% of the variance, indicating a minor role relative to judge-related effects.\"}]",1784205747,40,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":27},"judges-matter-more-than-papers-in-post-publication-research-assessment","",{"@graph":35,"@context":85},[36,53,68],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/judges-matter-more-than-papers-in-post-publication-research-assessment/85713/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-17","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What determines differences in research assessment outcomes according to the study?","Question",{"text":75,"@type":76},"Differences between evaluators (judges) explain substantially more variance in ratings than differences between the evaluated papers.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"How was the variance in research quality ratings analyzed?",{"text":80,"@type":76},"The study used multilevel models to partition rating variance into judge-related components and paper/journal-related components, including judge severity and attribute weighting.",{"name":82,"@type":73,"acceptedAnswer":83},"Did the study find evidence of directional bias such as author gender or global affiliation?",{"text":84,"@type":76},"Directional bias measures like author gender and global affiliation explained less than 1% of the variance, indicating a minor role relative to judge-related effects.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,119,122,127,130,134],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":45,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":45,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":45,"category_name":117,"show_sort_weight":28,"slug":118},7,"Healthcare","healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":120,"slug":121},30,"research-report",{"id":123,"doc_module":4,"doc_module_name":45,"category_name":124,"show_sort_weight":125,"slug":126},9,"Religion & Spirituality",20,"religion-spirituality",{"id":125,"doc_module":4,"doc_module_name":45,"category_name":128,"show_sort_weight":125,"slug":129},"World Cup","world-cup",{"id":131,"doc_module":4,"doc_module_name":45,"category_name":132,"show_sort_weight":131,"slug":133},10,"Lifestyle","lifestyle",{"id":135,"doc_module":4,"doc_module_name":45,"category_name":136,"show_sort_weight":106,"slug":137},19,"General","general"]