[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-seo-293109-105":3,"detail-sidebar-cat-0-en-105":81,"doc-detail-293109-en":131},{"code":4,"msg":5,"data":6},0,"ok",{"site_id":7,"language":8,"slug":9,"title":10,"keywords":11,"description":12,"schema_data":13,"social_meta":74,"head_meta":76,"extra_data":78,"updated_unix":80},105,"en","can-chatgpt-assess-human-personalities-a-general-evaluation-framework-293109","Can ChatGPT Assess Human Personalities? - A General Evaluation Framework","","Large Language Models, especially ChatGPT, have achieved strong performance across many tasks, yet their capacity for human-like psychology remains insufficiently studied. This paper introduces a general evaluation framework that uses Myers–Briggs Type Indicator (MBTI) tests to assess human personalities via LLMs. It designs unbiased MBTI prompts by random option permutation, averages results for impartial answers, and enables flexible subject replacement. It also reformulates instructions for correctness evaluation, and proposes metrics for consistency, robustness, and fairness, showing ChatGPT can deliver more consistent and fair assessments than InstructGPT despite reduced robustness to prompt bias.",{"@graph":14,"@context":73},[15,34,56],{"@type":16,"itemListElement":17},"BreadcrumbList",[18,23,27,31],{"item":19,"name":20,"@type":21,"position":22},"https://docshare.wps.com","Home","ListItem",1,{"item":24,"name":25,"@type":21,"position":26},"https://docshare.wps.com/document/","Document",2,{"item":28,"name":29,"@type":21,"position":30},"https://docshare.wps.com/document/research-report/","Research & Report",3,{"item":32,"name":10,"@type":21,"position":33},"https://docshare.wps.com/document/can-chatgpt-assess-human-personalities-a-general-evaluation-framework-293109/293109/",4,{"url":32,"name":10,"@type":35,"image":36,"author":41,"headline":10,"publisher":44,"fileFormat":47,"inLanguage":8,"description":12,"dateModified":48,"datePublished":49,"encodingFormat":47,"isAccessibleForFree":50,"interactionStatistic":51},"DigitalDocument",{"url":37,"@type":38,"width":39,"height":40},"https://docshare.wps.com/thumbnails/can-chatgpt-assess-human-personalities-a-general-evaluation-framework-293109/293109.png","ImageObject",300,407,{"name":42,"@type":43},"Kyle","Person",{"url":19,"name":45,"@type":46},"DocShare","Organization","application/pdf","2026-11-04","2026-09-17",true,{"@type":52,"interactionType":53,"userInteractionCount":55},"InteractionCounter",{"@type":54},"ViewAction",12,{"@type":57,"mainEntity":58},"FAQPage",[59,65,69],{"name":60,"@type":61,"acceptedAnswer":62},"What is the paper’s main goal?","Question",{"text":63,"@type":64},"To provide a generic evaluation framework that uses LLMs to assess human personalities based on MBTI tests.","Answer",{"name":66,"@type":61,"acceptedAnswer":67},"How does the framework make MBTI prompts more unbiased?",{"text":68,"@type":64},"It randomly permutes the answer options and averages the testing results to encourage more impartial responses.",{"name":70,"@type":61,"acceptedAnswer":71},"What evaluation metrics does the paper propose?",{"text":72,"@type":64},"Three metrics to measure consistency, robustness, and fairness of assessment results from state-of-the-art LLMs such as ChatGPT and GPT-4.","https://schema.org",{"og:url":32,"og:type":75,"og:title":10,"og:site_name":45,"og:description":12},"article",{"robots":77,"canonical":32},"index,follow",{"doc_id":79,"site_id":7},293109,1789659937,{"code":4,"msg":82,"data":83},"success",[84,88,92,96,101,106,111,115,120,123,127],{"id":22,"doc_module":4,"doc_module_name":25,"category_name":85,"show_sort_weight":86,"slug":87},"Story & Novel",90,"story-novel",{"id":26,"doc_module":4,"doc_module_name":25,"category_name":89,"show_sort_weight":90,"slug":91},"Literature",80,"literature",{"id":33,"doc_module":4,"doc_module_name":25,"category_name":93,"show_sort_weight":94,"slug":95},"Exam",70,"exam",{"id":97,"doc_module":4,"doc_module_name":25,"category_name":98,"show_sort_weight":99,"slug":100},5,"Comic",60,"comic",{"id":102,"doc_module":4,"doc_module_name":25,"category_name":103,"show_sort_weight":104,"slug":105},6,"Technology",50,"technology",{"id":107,"doc_module":4,"doc_module_name":25,"category_name":108,"show_sort_weight":109,"slug":110},7,"Healthcare",40,"healthcare",{"id":112,"doc_module":4,"doc_module_name":25,"category_name":29,"show_sort_weight":113,"slug":114},8,30,"research-report",{"id":116,"doc_module":4,"doc_module_name":25,"category_name":117,"show_sort_weight":118,"slug":119},9,"Religion & Spirituality",20,"religion-spirituality",{"id":118,"doc_module":4,"doc_module_name":25,"category_name":121,"show_sort_weight":118,"slug":122},"World Cup","world-cup",{"id":124,"doc_module":4,"doc_module_name":25,"category_name":125,"show_sort_weight":124,"slug":126},10,"Lifestyle","lifestyle",{"id":128,"doc_module":4,"doc_module_name":25,"category_name":129,"show_sort_weight":97,"slug":130},19,"General","general",{"code":4,"msg":82,"data":132},{"doc_id":79,"user_id":133,"nickname":42,"user_avatar":134,"doc_module":4,"category_id":112,"category_name":29,"doc_title":10,"doc_description":12,"doc_content":135,"file_id":136,"file_url":137,"file_type":138,"file_size":139,"view_count":55,"is_deleted":4,"is_public":22,"is_downloadable":22,"audit_status":22,"page_count":140,"language":141,"language_code":8,"site_id":7,"html_lang":8,"table_of_contents":142,"faqs":143,"seo_title":144,"seo_description":12,"update_tm":80,"read_time":145,"preview_page_count":4},3985741905716,"https://ap-avatar.wpscdn.com/davatar_994ba38a5ba835b3df7d355c54d3ed8d","Can ChatGPT Assess Human Personalities?  \nA General Evaluation Framework  \nHaocong Rao 1 ,2 Cyril Leung2 ,3 Chunyan Miao 1 ,2 ∗  \n1 School of Computer Science and Engineering, Nanyang Technological University, Singapore  \n2LILY Research Centre, Nanyang Technological University, Singapore  \n3Department of Electrical and Computer Engineering, The University of British Columbia, Canada {haocong001,[ascymiao}@ntu.edu.sg {cleung}@ece.ubc.ca](ascymiao}@ntu.edu.sg {cleung}@ece.ubc.ca)  \narXiv :2303 .01248v3 [ cs .CL] 13 Oct 2023  \nAbstract  \nLarge Language Models (LLMs) especially ChatGPT have produced impressive results in various areas, but their potential human-like psychology is still largely unexplored. Existing works study the virtual personalities of LLMs but rarely explore the possibility of analyzing human personalities via LLMs. This paper presents a generic evaluation framework for LLMs to assess human personalities based on Myers–Briggs Type Indicator (MBTI) tests.  \nSpecifically, we first devise unbiased prompts by randomly permuting options in MBTI questions and adopt the average testing result to encourage more impartial answer generation.  \nThen, we propose to replace the subject in question statements to enable flexible queries and assessments on different subjects from LLMs. Finally, we re-formulate the question instructions in a manner of correctness evaluation to facilitate LLMs to generate clearer responses. The proposed framework enables LLMs to flexibly assess personalities of different groups of people. We further propose three evaluation metrics to measure the consistency, robustness, and fairness of assessment results from state-ofthe-art LLMs including ChatGPT and GPT-4 . Our experiments reveal ChatGPT’s ability to assess human personalities, and the average results demonstrate that it can achieve more consistent and fairer assessments in spite of lower robustness against prompt biases compared with InstructGPT†.  \n1 Introduction  \nPre-trained Large Language Models (LLMs) have been widely used in many applications including translation, storytelling, and chatbots (Devlin et al., 2019 ; Raffel et al., 2020 ; Yang et al., 2022 ; Yuan et al., 2022 ; Ouyang et al., 2022 ; Bubeck et al.,  \n* Corresponding author  \n†Our codes are available at [https://github.com/Kali](https://github.com/Kali)Hac/ChatGPT-MBTI.  \n2023) . ChatGPT (Ouyang et al., 2022) and its enhanced version GPT-4 are currently recognized as the most capable chatbots, which can perform context-aware conversations, challenge incorrect premises, and reject inappropriate requests with a vast knowledge base and human-centered finetuning. These advantages make them well-suited for a variety of real-world scenarios such as business consultation and educational services (Zhai, 2022 ; van Dis et al., 2023 ; Bubeck et al., 2023) .  \nRecent studies have revealed that LLMs may possess human-like self-improvement and reasoning characteristics (Huang et al., 2022 ; Bubeck et al., 2023) . The latest GPT series can pass over 90% of Theory of Mind (ToM) tasks with strong analysis and decision-making capabilities (Kosinski, 2023 ; Zhuo et al., 2023 ; Moghaddam and Honey, 2023) . In this context, LLMs are increasingly assumed to have virtual personalities and psychologies, which plays an essential role in guiding their responsesand interaction patterns (Jiang et al., 2022) . Based on this assumption, a few works (Li et al., 2022 ; Jiang et al., 2022 ; Karra et al., 2022 ; Caron and Srivastava, 2022 ; Miotto et al., 2022) apply psychological tests such as Big Five Factors (Digman, 1990) to evaluate their pseudo personalities (e.g. , behavior tendency), so as to detect societal and ethical risks (e.g. , racial biases) in their applications. Although existing works have investigated the personality traits of LLMs, they rarely explored whether LLMs can assess human personalities. This open problem can be the key to verifying the ability of LLMs to perform psychological ","cbCaiqQSFlScNryQ","https://ap.wps.com/l/cbCaiqQSFlScNryQ","pdf",1606174,21,"English","# Abstract\n# Introduction\n## Motivation and related work\n## Proposed evaluation framework\n### Unbiased prompts\n### Subject-replaced queries\n### Correctness-evaluated instructions","[{\"question\":\"What is the paper’s main goal?\",\"answer\":\"To provide a generic evaluation framework that uses LLMs to assess human personalities based on MBTI tests.\"},{\"question\":\"How does the framework make MBTI prompts more unbiased?\",\"answer\":\"It randomly permutes the answer options and averages the testing results to encourage more impartial responses.\"},{\"question\":\"What evaluation metrics does the paper propose?\",\"answer\":\"Three metrics to measure consistency, robustness, and fairness of assessment results from state-of-the-art LLMs such as ChatGPT and GPT-4.\"}]","Can ChatGPT Assess Human Personalities? - A General Evaluation Framework | PDF",53]