[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-seo-293654-105":3,"detail-sidebar-cat-0-en-105":81,"doc-detail-293654-en":131},{"code":4,"msg":5,"data":6},0,"ok",{"site_id":7,"language":8,"slug":9,"title":10,"keywords":11,"description":12,"schema_data":13,"social_meta":74,"head_meta":76,"extra_data":78,"updated_unix":80},105,"en","rethinking-role-playing-evaluation-anonymous-benchmarking-and-a-systematic-study-of-personality-effects","Rethinking Role-Playing Evaluation - Anonymous Benchmarking and A Systematic Study of Personality Effects","","Large Language Models (LLMs) show strong capability in role-playing agents, but existing evaluation often depends on well-known fictional characters. This creates a risk that models exploit memorized identity cues rather than true role-playing ability, leading to degraded performance on unseen or out-of-distribution personas. The study introduces an anonymized evaluation protocol that decouples role proficiency from character recognition and tests personality-augmentation strategies to improve fidelity, reporting consistent gains and more equitable robustness.",{"@graph":14,"@context":73},[15,34,56],{"@type":16,"itemListElement":17},"BreadcrumbList",[18,23,27,31],{"item":19,"name":20,"@type":21,"position":22},"https://docshare.wps.com","Home","ListItem",1,{"item":24,"name":25,"@type":21,"position":26},"https://docshare.wps.com/document/","Document",2,{"item":28,"name":29,"@type":21,"position":30},"https://docshare.wps.com/document/research-report/","Research & Report",3,{"item":32,"name":10,"@type":21,"position":33},"https://docshare.wps.com/document/rethinking-role-playing-evaluation-anonymous-benchmarking-and-a-systematic-study-of-personality-effects/293654/",4,{"url":32,"name":10,"@type":35,"image":36,"author":41,"headline":10,"publisher":44,"fileFormat":47,"inLanguage":8,"description":12,"dateModified":48,"datePublished":49,"encodingFormat":47,"isAccessibleForFree":50,"interactionStatistic":51},"DigitalDocument",{"url":37,"@type":38,"width":39,"height":40},"https://docshare.wps.com/thumbnails/rethinking-role-playing-evaluation-anonymous-benchmarking-and-a-systematic-study-of-personality-effects/293654.png","ImageObject",300,407,{"name":42,"@type":43},"Aladdin","Person",{"url":19,"name":45,"@type":46},"DocShare","Organization","application/pdf","2026-11-06","2026-09-17",true,{"@type":52,"interactionType":53,"userInteractionCount":55},"InteractionCounter",{"@type":54},"ViewAction",11,{"@type":57,"mainEntity":58},"FAQPage",[59,65,69],{"name":60,"@type":61,"acceptedAnswer":62},"What problem does the paper identify in current role-playing evaluation?","Question",{"text":63,"@type":64},"Current frameworks rely heavily on well-known fictional characters, which may let models use internal memorized cues instead of genuine role-playing ability.","Answer",{"name":66,"@type":61,"acceptedAnswer":67},"How does the proposed evaluation protocol address character recognition bias?",{"text":68,"@type":64},"It conceals character identities so LLMs must rely on provided character descriptions rather than pretraining-based prior knowledge.",{"name":70,"@type":61,"acceptedAnswer":71},"What technique is investigated to improve performance under anonymized settings?",{"text":72,"@type":64},"The paper investigates personality augmentation, including using personality traits from The Personality Database (PDB) and model-generated personalities to guide role play.","https://schema.org",{"og:url":32,"og:type":75,"og:title":10,"og:site_name":45,"og:description":12},"article",{"robots":77,"canonical":32},"index,follow",{"doc_id":79,"site_id":7},293654,1789667089,{"code":4,"msg":82,"data":83},"success",[84,88,92,96,101,106,111,115,120,123,127],{"id":22,"doc_module":4,"doc_module_name":25,"category_name":85,"show_sort_weight":86,"slug":87},"Story & Novel",90,"story-novel",{"id":26,"doc_module":4,"doc_module_name":25,"category_name":89,"show_sort_weight":90,"slug":91},"Literature",80,"literature",{"id":33,"doc_module":4,"doc_module_name":25,"category_name":93,"show_sort_weight":94,"slug":95},"Exam",70,"exam",{"id":97,"doc_module":4,"doc_module_name":25,"category_name":98,"show_sort_weight":99,"slug":100},5,"Comic",60,"comic",{"id":102,"doc_module":4,"doc_module_name":25,"category_name":103,"show_sort_weight":104,"slug":105},6,"Technology",50,"technology",{"id":107,"doc_module":4,"doc_module_name":25,"category_name":108,"show_sort_weight":109,"slug":110},7,"Healthcare",40,"healthcare",{"id":112,"doc_module":4,"doc_module_name":25,"category_name":29,"show_sort_weight":113,"slug":114},8,30,"research-report",{"id":116,"doc_module":4,"doc_module_name":25,"category_name":117,"show_sort_weight":118,"slug":119},9,"Religion & Spirituality",20,"religion-spirituality",{"id":118,"doc_module":4,"doc_module_name":25,"category_name":121,"show_sort_weight":118,"slug":122},"World Cup","world-cup",{"id":124,"doc_module":4,"doc_module_name":25,"category_name":125,"show_sort_weight":124,"slug":126},10,"Lifestyle","lifestyle",{"id":128,"doc_module":4,"doc_module_name":25,"category_name":129,"show_sort_weight":97,"slug":130},19,"General","general",{"code":4,"msg":82,"data":132},{"doc_id":79,"user_id":133,"nickname":42,"user_avatar":134,"doc_module":4,"category_id":112,"category_name":29,"doc_title":10,"doc_description":12,"doc_content":135,"file_id":136,"file_url":137,"file_type":138,"file_size":139,"view_count":55,"is_deleted":4,"is_public":22,"is_downloadable":22,"audit_status":22,"page_count":140,"language":141,"language_code":8,"site_id":7,"html_lang":8,"table_of_contents":142,"faqs":143,"seo_title":144,"seo_description":12,"update_tm":80,"read_time":145,"preview_page_count":4},2336478503145,"https://ap-avatar.wpscdn.com/davatar_276721f389ce27ea32af1340a28f341c","Rethinking Role-Playing Evaluation:  \nAnonymous Benchmarking and A Systematic Study of Personality Effects  \nJi-Lun Peng⋆† Yun-Nung Chen⋆  \n⋆National Taiwan University, Taipei, Taiwan  \n†Academia Sinica, Taipei, Taiwan  \n[r13946006@ntu.edu.tw](r13946006@ntu.edu.tw) [y.v.chen@ieee.org](y.v.chen@ieee.org)  \narXiv :2603 .03915v2 [ cs .CL] 29 Jun 2026  \nAbstract  \nLarge Language Models (LLMs) have shown remarkable potential in developing role-playing agents (RPAs) . However, current evaluation frameworks rely heavily on well-known fictional characters, raising a critical concern:  \nmodels may be leveraging their internal training memory of these characters rather than demonstrating role-playing capabilities. This reliance often leads to significant performance degradation when RPAs encounter unseen or out-of-distribution personas. To address this, we propose a more rigorous evaluation protocol designed to decouple role-playing proficiency from character recognition. Our experiments across multiple benchmarks demonstrate that anonymizing characters degrades performance, confirming that name exposure provides implicit cues that mask a model’s true capability. To mitigate this, we investigate diverse personality augmentation as a method to enhance role fidelity in anonymous settings. We systematically analyze the impact of various personalitydescription methods on agent behavior and consistency. Our results show that incorporating personality information consistently improves RPA performance. This work establishes amore equitable evaluation standard and validates a scalable, personality-enhanced framework for constructing robust RPAs.1  \n1 Introduction  \nLLMs have demonstrated strong performance across diverse tasks (Peng et al., 2024) . A recent and significant application involves their use as RPAs, where they simulate specific professions (Wang et al., 2023) and characters (Shao et al., 2023) . This approach enhances the human-likeness of interactions, with character simulation being a major application area (Tseng et al., 2024 ; Chenet al., 2024 ; He et al., 2025 ; Wu et al., 2025) .  \n1Code: [https://github.com/MiuLab/RolePlayBench](https://github.com/MiuLab/RolePlayBench)  \nCurrent research on role-playing agents primarily focuses on simulating fictional characters, such as those from novels and films, prompting language models to generate text consistent with a predefined character style (Yu et al., 2025 ; Qin et al., 2025 ; Zhang et al., 2025c ; Wang et al., 2025a) . This line of work includes the construction of various benchmark datasets for evaluating role-playing capabilities (Wang et al., 2025a ; Dai et al., 2024 ; Wu et al., 2025), as well as methodological advancements designed to enhance LLMs’ role-playing performance (Gao et al., 2025 ; Yang et al., 2025 ; Ye et al., 2025 ; Liu et al., 2025) . Enabling LLMs to simulate fictional characters has practical applications in avatar-based interactions2 and interactive gaming environments (Zhang et al., 2025a), thereby extending virtual characters beyond their original narrative contexts.  \nIn addition, several studies have explored the simulation of real individuals, including tasks such as survey completion (Park et al., 2024), social media post generation (Huang, 2024), and message response imitation (Shi et al., 2025) . However, evaluation methodologies for LLMs impersonating lesser-known real humans remain limited, largely due to the difficulty of collecting high-quality, large-scale data about lesser-known real individuals. Moreover, well-known fictional characters are often well-represented in LLM pretraining corpora, granting models substantial prior knowledge about their personas. Consequently, high proficiency in role-playing these famous figures does not necessarily generalize to the simulation of lesser-known individuals, as models are unfamiliar with the these people.  \nTo address this imbalance in prior knowledge, we propose an anonymized role-playing evaluatio","cbCairwkwK8Bei43","https://ap.wps.com/l/cbCairwkwK8Bei43","pdf",891679,14,"English","# Abstract\n# Introduction\n## Role-playing agents and existing benchmarks\n## Limitations of memorized character identities\n## Proposed anonymized evaluation protocol\n## Personality augmentation for anonymous role-play\n# Figure 1 Framework overview","[{\"question\":\"What problem does the paper identify in current role-playing evaluation?\",\"answer\":\"Current frameworks rely heavily on well-known fictional characters, which may let models use internal memorized cues instead of genuine role-playing ability.\"},{\"question\":\"How does the proposed evaluation protocol address character recognition bias?\",\"answer\":\"It conceals character identities so LLMs must rely on provided character descriptions rather than pretraining-based prior knowledge.\"},{\"question\":\"What technique is investigated to improve performance under anonymized settings?\",\"answer\":\"The paper investigates personality augmentation, including using personality traits from The Personality Database (PDB) and model-generated personalities to guide role play.\"}]","Rethinking Role-Playing Evaluation - Anonymous Benchmarking and A Systematic Study of Personality Effects | PDF",35]