Using LLMs to Identify Features of Personal and Professional Skills in an Open-Response Situational Judgment Test

Oct 3, 2026

Academic programs increasingly value personal and professional skills alongside technical expertise, yet scalable, valid measurement systems remain limited. This work examines how large language models can extract construct-relevant features from open-response situational judgment test (SJT) answers, addressing prior NLP scoring approaches that struggled with construct validity. Using the Casper SJT, the study demonstrates the approach’s effectiveness and provides a foundation for automated scoring of these skills.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages10

Reading Time25 min

AccessFree to download

Background

Skills assessment in admissions

Formats of SJTs

Automated scoring with NLP

Method and evaluation

Feature extraction with LLMs

Casper SJT results

Conclusion

Using LLMs to identify features of personal and professional skills in an open-response situational judgment test Cole Walsh, Rodica Ivan, Muhammad Zafar Iqbal, and Colleen Robb Acuity Insights Inc., Toronto, ON, Canada arXiv :2507 . 1388 1v 1 [ cs .CL] 18 Jul 2025 Abstract Academic programs are increasingly recognizing the importance of personal and professional skills and their critical role alongside technical expertise in preparing students for future success in diverse career paths. With this growing demand comes the need for scalable systems to measure, evaluate, and develop these skills. Situational Judgment Tests (SJTs) offer one potential avenue for measuring these skills in a standardized and reliable way, but open-response SJTs have traditionally relied on trained human raters for evaluation, presenting operational challenges to delivering SJTs at scale. Past attempts at developing NLP-based scoring systems for SJTs have fallen short due to issues with construct validity of these systems. In this article, we explore a novel approach to extracting construct-relevant features from SJT responses using large language models (LLMs) . We use the Casper SJT to demonstrate the efficacy of this approach. This study sets the foundation for future developments in automated scoring for personal and professional skills. 1 Background A longstanding challenge in academia is selecting qualified and professional candidates from a larger applicant pool for professional training programs. Decision makers have traditionally relied on measures of hard skills and cognitive ability to make these decisions (Eva et al., 2009), often relying on grade point average (GPA) and standardized tests such as the Scholastic Aptitude Test (SAT), Graduate Record Exam (GRE), Medical College Admission Test (MCAT), and Graduate Management Admission Test (GMAT) . Personal and professional skills such as communication, teamwork, problem-solving, and critical thinking, although recognized as predictive of future success in education and industry (Heckman and Kautz, 2012), have been more difficult to measure for a number of reasons including lack of standardization and scalability (Patterson et al., 2016) . Admissions committees have commonly used reference letters, personal essays, and interviews as a proxy for applicants’ personal and professional skills, but these processes do not meet the psychometric standards that we would expect from tools used in high-stakes decision-making (Kuncel et al., 2014 ; Patterson et al., 2016) . Additionally, as the adoption of generative AI spreads, there are increased concerns about the authenticity of reference letters and personal essays (Chen et al., 2024), further exacerbating the need for valid and reliable tools to measure personal and professional skills. Recognizing the limitations of other admissions tools (i.e., reference letters, personal essays) (Patterson et al., 2016), higher education programs have been increasingly turning to a more reliable and standardized tool, Situational Judgment Tests (SJTs), to assess applicants’ personal and professional skills as part of their admissions process (Webster et al., 2020 ; Nadmilail et al., 2023) . Though they may be delivered in different formats, including fixed-response and open-response, SJTs generally involve simulated situations and questions designed to elicit how a respondent would likely react in the situation (Lievens, 2013) . Fixedresponse SJTs typically require respondents to select or rank possible actions based on their effectiveness in a given situation and show stronger relationships with measures of cognitive ability, rather than personal or professional skills (McDaniel et al., 2007) . Open-response SJTs, on the other hand, are more conducive to measuring behavioral tendencies (i.e., how the respondent would likely react in the given situation) and tend to show stronger relationships with personal and professional skills relati

FAQs

Why are personal and professional skills difficult to measure at scale in admissions?

They are often assessed through unstandardized tools like reference letters, essays, and interviews, and open-response SJTs typically require extensive human rater training to score reliably.

What problem do prior NLP-based scoring systems face for open-response SJTs?

They have fallen short due to construct validity issues, meaning the scoring systems did not validly reflect the intended skill constructs measured by the tests.

How does the paper improve automated evaluation of open-response SJTs?

It explores extracting construct-relevant features from SJT responses using large language models, demonstrated with the Casper SJT to support effective automated scoring.

Oct 3, 2026

Upload by RuangKosong