The Impact of Large Language Models on Applicants’ Cover Letter Scores - and the Moderation Effect of Language Nativity

Sep 13, 2026

The study examines how large language model (LLM) use in applicants’ cover letter writing affects recruiter evaluations and whether language nativity changes this influence. Using an experimental design with two-way repeated measures ANOVA, it compares recruiter ratings across cover letters created or enhanced with ChatGPT versus none, for native and English-as-a-second-language groups. Recruiter assessments are collected via a Hireability Index using Qualtrics, with 46 responses analyzed. Results show higher scores for ChatGPT-used letters, while the language proficiency interaction is insignificant.

LanguageEnglish

CategoryPresentations

FormatPDF

Pages32

Reading Time11 min

AccessFree to download

Abstract

Introduction

The Impact of Large Language Models on Applicants’ Cover Letter Scores and the Moderation Effect of Language Nativity Ceren Kalkir SNR: 2100029 Tilburg University M.Sc. Work and Organisational Psychology 2023/2024 Supervisor: Dr. Djurre Holtrop 2nd reader: Dr. Bence Bago Abstract The use of the Large Language Model in the cover letter writing process by candidates increased the ethical considerations in the recruitment process. However, the outcomes ofthe use of the Large Language Model (LLM) by the candidates in recruiters’ evaluations are limited. Additionally, LLM usage may challenge the positive effect of language nativity on the recruiter evaluations. Therefore, this research aims to answer the following: How does the use of various types of LLMs impact recruiter evaluations of Cover Letters, and is this impact influenced by language nativity? This study utilises an experimental design and two-way repeated measures ANOVA to assess the recruiters’ evaluations for various cover letters representing one unique combination of Chat GPT usage types (within-subjects) by native and English as a second language groups (between-subjects) . Candidates can create a new cover letter or enhance their prewritten cover letter in the cover letter writing process with the utilisation of LLMs. Recruiter evaluations were measured using the Hireability Index. Qualtrics survey tool used to collect the recruiters’evaluations on cover letters. Forty-six final responses were included in the analysis. Recruiters’cover letter evaluations increased from no Chat GPT-used cover letters to Chat GPT enhanced and Chat GPT created cover letters. However, the recruiters’ evaluations did not show a significant difference between Chat GPT enhanced and Chat GPT created cover letters. The interaction effect of language proficiency and Chat GPT usage was insignificant. The main effect of language proficiency on cover letter scores was significant, with higher cover letter scores for Native speakers than for English as a second language (ESL) speakers. Post-hoc analysis revealed that recruiters’ evaluations are higher for native speaker candidates than ESL group candidates when no LLM is utilised. The difference between ESL and native groups is insignificant when candidates utilise LLMs by creating or enhancing their cover letters. Introduction The cover letter is often the initial business communication from a candidate. The cover letters written by a candidate affect the organisation’s decision, which leads to the invitation of the right candidate for an interview (Watts, 2015, p. 10) . The advancements in LLMs have affected the candidates’ application process for their new jobs (Nuzula & Amri, 2023) . Nuzula and Amri (2023) stated that the utilisation ofLLMs by the candidates is ethically acceptable for some tasks, such as preparation for an interview and identification of candidates’ skills for a specific position. However, they identified that some tasks raise ethical considerations, such as utilising LLMs in the cover letter writing process. Utilisation of LLMs by the applicants in the cover letter writing process may lead to the invitation of the wrong candidate for an interview, as the cover letter is not written by the candidate but instead by an LLM. Additionally, LLMs are also expected to challenge the previously known positive effect of the applicant’s language nativity on cover letter evaluations of the recruiters (Carlsson et al., 2023) . Therefore, more insights are needed on the recruiter’s evaluations between human written and Chat GPT utilised cover letters to eliminate the risk of invitation of the wrong candidate for an interview. This research is theoretically relevant as an understanding of LLMs and their outcomes in the recruitment field is limited. First, before raising the ethical considerations, it is essential to analyse whether the utilisation of LLMs in the cover letter writing process leads to better evaluati

FAQs

What is the main research question of the study?

How LLM use in cover letter writing affects recruiter evaluations, and whether this impact is moderated by language nativity/language proficiency.

How were recruiter evaluations measured?

Recruiter evaluations were measured using the Hireability Index, with data collected through the Qualtrics survey tool.

What did the results show about ChatGPT-created versus ChatGPT-enhanced cover letters?

Recruiter evaluations increased from no-ChatGPT letters to both ChatGPT-enhanced and ChatGPT-created letters, but the difference between enhanced and created letters was not significant.

Sep 13, 2026

Uploaded by Berry Peter