[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-seo-293090-105":3,"detail-sidebar-cat-0-en-105":81,"doc-detail-293090-en":130},{"code":4,"msg":5,"data":6},0,"ok",{"site_id":7,"language":8,"slug":9,"title":10,"keywords":11,"description":12,"schema_data":13,"social_meta":74,"head_meta":76,"extra_data":78,"updated_unix":80},105,"en","revisiting-the-reliability-of-psychological-scales-on-large-language-models","Revisiting the Reliability of Psychological Scales on Large Language Models","","Recent research examines Large Language Models (LLMs) through psychological lenses, motivating personality testing for these systems. Yet the validity of psychological scales originally designed for humans remains contested. This study assesses whether LLMs exhibit consistent personality traits when administered the Big Five Inventory, focusing on reliability under multiple experimental conditions. Analysis across 2,500 settings per model (GPT-3.5, GPT-4, Gemini-Pro, LLaMA-3.1) shows stable response consistency, supporting the scales’ reliability. Results also suggest GPT-3.5 can be prompted to emulate diverse personalities.",{"@graph":14,"@context":73},[15,34,56],{"@type":16,"itemListElement":17},"BreadcrumbList",[18,23,27,31],{"item":19,"name":20,"@type":21,"position":22},"https://docshare.wps.com","Home","ListItem",1,{"item":24,"name":25,"@type":21,"position":26},"https://docshare.wps.com/document/","Document",2,{"item":28,"name":29,"@type":21,"position":30},"https://docshare.wps.com/document/research-report/","Research & Report",3,{"item":32,"name":10,"@type":21,"position":33},"https://docshare.wps.com/document/revisiting-the-reliability-of-psychological-scales-on-large-language-models/293090/",4,{"url":32,"name":10,"@type":35,"image":36,"author":41,"headline":10,"publisher":44,"fileFormat":47,"inLanguage":8,"description":12,"dateModified":48,"datePublished":49,"encodingFormat":47,"isAccessibleForFree":50,"interactionStatistic":51},"DigitalDocument",{"url":37,"@type":38,"width":39,"height":40},"https://docshare.wps.com/thumbnails/revisiting-the-reliability-of-psychological-scales-on-large-language-models/293090.png","ImageObject",300,407,{"name":42,"@type":43},"Caleb Sterling","Person",{"url":19,"name":45,"@type":46},"DocShare","Organization","application/pdf","2026-10-30","2026-09-17",true,{"@type":52,"interactionType":53,"userInteractionCount":55},"InteractionCounter",{"@type":54},"ViewAction",20,{"@type":57,"mainEntity":58},"FAQPage",[59,65,69],{"name":60,"@type":61,"acceptedAnswer":62},"What problem does the paper address about applying psychological scales to LLMs?","Question",{"text":63,"@type":64},"The paper examines whether psychological scales designed for humans can be reliably used with LLMs, focusing on whether LLMs show consistent personality traits under different conditions.","Answer",{"name":66,"@type":61,"acceptedAnswer":67},"How is reliability evaluated in this study?",{"text":68,"@type":64},"Reliability is assessed by analyzing response consistency on the Big Five Inventory across many experimental settings and condition variations for each model.",{"name":70,"@type":61,"acceptedAnswer":71},"What experimental evidence does the paper provide?",{"text":72,"@type":64},"The study analyzes 2,500 settings per model, including GPT-3.5, GPT-4, Gemini-Pro, and LLaMA-3.1, and finds that responses to the Big Five Inventory are sufficiently consistent for multiple LLMs.","https://schema.org",{"og:url":32,"og:type":75,"og:title":10,"og:site_name":45,"og:description":12},"article",{"robots":77,"canonical":32},"index,follow",{"doc_id":79,"site_id":7},293090,1789659491,{"code":4,"msg":82,"data":83},"success",[84,88,92,96,101,106,111,115,119,122,126],{"id":22,"doc_module":4,"doc_module_name":25,"category_name":85,"show_sort_weight":86,"slug":87},"Story & Novel",90,"story-novel",{"id":26,"doc_module":4,"doc_module_name":25,"category_name":89,"show_sort_weight":90,"slug":91},"Literature",80,"literature",{"id":33,"doc_module":4,"doc_module_name":25,"category_name":93,"show_sort_weight":94,"slug":95},"Exam",70,"exam",{"id":97,"doc_module":4,"doc_module_name":25,"category_name":98,"show_sort_weight":99,"slug":100},5,"Comic",60,"comic",{"id":102,"doc_module":4,"doc_module_name":25,"category_name":103,"show_sort_weight":104,"slug":105},6,"Technology",50,"technology",{"id":107,"doc_module":4,"doc_module_name":25,"category_name":108,"show_sort_weight":109,"slug":110},7,"Healthcare",40,"healthcare",{"id":112,"doc_module":4,"doc_module_name":25,"category_name":29,"show_sort_weight":113,"slug":114},8,30,"research-report",{"id":116,"doc_module":4,"doc_module_name":25,"category_name":117,"show_sort_weight":55,"slug":118},9,"Religion & Spirituality","religion-spirituality",{"id":55,"doc_module":4,"doc_module_name":25,"category_name":120,"show_sort_weight":55,"slug":121},"World Cup","world-cup",{"id":123,"doc_module":4,"doc_module_name":25,"category_name":124,"show_sort_weight":123,"slug":125},10,"Lifestyle","lifestyle",{"id":127,"doc_module":4,"doc_module_name":25,"category_name":128,"show_sort_weight":97,"slug":129},19,"General","general",{"code":4,"msg":82,"data":131},{"doc_id":79,"user_id":132,"nickname":42,"user_avatar":133,"doc_module":4,"category_id":112,"category_name":29,"doc_title":10,"doc_description":12,"doc_content":134,"file_id":135,"file_url":136,"file_type":137,"file_size":138,"view_count":55,"is_deleted":4,"is_public":22,"is_downloadable":22,"audit_status":22,"page_count":139,"language":140,"language_code":8,"site_id":7,"html_lang":8,"table_of_contents":141,"faqs":142,"seo_title":143,"seo_description":12,"update_tm":80,"read_time":99,"preview_page_count":4},962084925290,"https://ap-avatar.wpscdn.com/davatar_085a072bc5b1113ac321206ff7593b45","arXiv :2305 . 19926v 5 [ cs .CL] 4 Oct 2024  \nRevisiting the Reliability of Psychological Scales on Large Language Models   \nREVISITING THE RELIABILITY OF PSYCHOLOGICAL SCALES ON LARGE LANGUAGE MODELS  \nJen-tse Huang 1 ,2 , Wenxiang Jiao2 , Man Ho Lam 1 , Eric John Li 1 , Wenxuan Wang 1 ,2 Michael R. Lyu 1 1Department of Computer Science and Engineering, The Chinese University of Hong Kong  \n{jthuang,wxwang,[lyu](lyu}@cse.cuhk.edu.hk {mhlam)[}](lyu}@cse.cuhk.edu.hk {mhlam)[@cse.cuhk.edu.hk](lyu}@cse.cuhk.edu.hk {mhlam)[ {](lyu}@cse.cuhk.edu.hk {mhlam)[mhlam](lyu}@cse.cuhk.edu.hk {mhlam) ,[ejli](ejli}@link.cuhk.edu.hk)[}](ejli}@link.cuhk.edu.hk)[@link.cuhk.edu.hk](ejli}@link.cuhk.edu.hk)[ ](ejli}@link.cuhk.edu.hk)[2](2 Tencent AI Lab {joelwxjiao}@tencent.com)[ Tencent AI Lab](2 Tencent AI Lab {joelwxjiao}@tencent.com)[ {](2 Tencent AI Lab {joelwxjiao}@tencent.com)[joelwxjiao](2 Tencent AI Lab {joelwxjiao}@tencent.com)[}](2 Tencent AI Lab {joelwxjiao}@tencent.com)[@tencent.com](2 Tencent AI Lab {joelwxjiao}@tencent.com)  \n\n|  | |\n| --- | --- |\n| \u003Cbr>\u003Cbr>\u003Cbr>\u003Cbr>\u003Cbr>\u003Cbr>\u003Cbr>\u003Cbr>\u003Cbr>|  |\n|  | \u003Cbr>\u003Cbr>(b) Conscientiousness\u003Cbr>\u003Cbr>|\n|  |  |\n|  | (c) Extraversion\u003Cbr>|\n|  | \u003Cbr>\u003Cbr>\u003Cbr>\u003Cbr>\u003Cbr>\u003Cbr>\u003Cbr>\u003Cbr>\u003Cbr>|\n|  | (d) Agreeableness\u003Cbr>\u003Cbr>|\n|  | \u003Cbr>\u003Cbr>\u003Cbr>\u003Cbr>\u003Cbr>\u003Cbr>\u003Cbr>\u003Cbr>\u003Cbr>|\n|  | \u003Cbr>(e) Neuroticism\u003Cbr>|\n\nFigure 1: Biweekly measurements starting from mid-September 2023 to late-January 2024 of the BFI on GPT-3.5-Turbo. The model experienced two different versions (0613, 1106) during this period. The shadow represents the standard deviation (±Std) .  \nABSTRACT  \nRecent research has focused on examining Large Language Models’(LLMs) characteristics from a psychological standpoint, acknowledging the necessity of understanding their behavioral characteristics. The administration of personality tests to LLMs has emerged as a noteworthy area in this context. However, the suitability of employing psychological scales, initially devised for humans, on LLMs is a matter of ongoing debate. Our study aims to determine the reliability of applying personality assessments to LLMs, explicitly investigating whether LLMs demonstrate consistent personality traits. Analysis of 2,500 settings per model, including GPT-3.5, GPT-4, Gemini-Pro, and LLaMA-3.1, reveals that various LLMs show consistency in responses to the Big Five Inventory, indicating a satisfactory level of reliability. Furthermore, our research explores the potential of GPT-3.5 to emulate diverse personalities and represent various groups—a capability increasingly sought after in social sciences for substituting human participants with LLMs to  \nreduce costs. Our findings reveal that LLMs have the potential to represent differ ent personalities with specific prompt instructions.  \n∗Wenxuan Wang is the corresponding author.  \n1 INTRODUCTION  \nThe recent emergence of Large Language Models (LLMs) marks a significant advancement in the field of Artificial Intelligence (AI), showcasing its abilities in various natural language processing tasks, including text translation (Jiao et al., 2023), sentence revision (Wu et al., 2023), program repair (Fan et al., 2023), and program testing (Deng et al., 2023) . Furthermore, LLM applications extend beyond computer science, enhancing fields such as clinical medicine (Cascella et al., 2023), legal advice (Deroy et al., 2023), and education (Dai et al., 2023) . Currently, LLMs are catalyzing a paradigm shift in human-computer interaction, revolutionizing how individuals engage with computational systems. With the integration of LLMs, computers have transcended their traditional role as tools to become assistants, establishing a symbiotic relationship with users. Thus, the focus of research extends beyond assessing LLM performance to understanding their behaviors from a psychological perspective. Huang et al. (2024b) highlights the significance of psychological analysis on LLMs in developing AI assistants that are more human-like, empathetic, and eng","cbCaifloQTbA8pcY","https://ap.wps.com/l/cbCaifloQTbA8pcY","pdf",8606959,24,"English","# Abstract\n# 1 Introduction\n## Background and motivation\n## Reliability of psychological scales on LLMs","[{\"question\":\"What problem does the paper address about applying psychological scales to LLMs?\",\"answer\":\"The paper examines whether psychological scales designed for humans can be reliably used with LLMs, focusing on whether LLMs show consistent personality traits under different conditions.\"},{\"question\":\"How is reliability evaluated in this study?\",\"answer\":\"Reliability is assessed by analyzing response consistency on the Big Five Inventory across many experimental settings and condition variations for each model.\"},{\"question\":\"What experimental evidence does the paper provide?\",\"answer\":\"The study analyzes 2,500 settings per model, including GPT-3.5, GPT-4, Gemini-Pro, and LLaMA-3.1, and finds that responses to the Big Five Inventory are sufficiently consistent for multiple LLMs.\"}]","Revisiting the Reliability of Psychological Scales on Large Language Models | PDF"]