[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-123017-en":3,"doc-seo-123017-105":30,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":4,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":27,"seo_description":14,"update_tm":28,"read_time":29},123017,8796095462418,"Noah","https://ap-avatar.wpscdn.com/avatar/80000253c1241d02b47?x-image-process=image/resize,m_fixed,w_180,h_180&k=1778826106357471780",8,"Research & Report","A Quantitative Machine Learning Approach to Evaluating Letters of Recommendation","Letters of Recommendation (LOR) are central to undergraduate and graduate admissions, yet fair assessment is difficult because applicant–recommender relationships vary, standardized criteria are limited, and review resources are constrained. This work proposes three evaluation criteria—relevance, specificity, and positivity—and manually rates about 4,000 LORs using study-developed guidelines. Natural language processing and machine learning models are then trained to predict these ratings from the LOR text. The results support objective, automated LOR evaluation and enable selective review when staffing is limited.","Proceedings of the 57th Hawaii International Conference on System Sciences | 2024  \nA Quantitative Machine Learning Approach to Evaluating Letters of  \nRecommendation  \nYijun Zhao Fordham University  [yzhao11@fordham.edu](yzhao11@fordham.edu)  \nEllise Parnoff Fordham University [eparnoff@fordham.edu](eparnoff@fordham.edu)  \nTianyu Wang Fordham University  [twang183@fordham.edu](twang183@fordham.edu)  \nSiyi He Fordham University  [she81@fordham.edu](she81@fordham.edu)  \nDouglas Mensah Fordham University [dmensah4@fordham.edu](dmensah4@fordham.edu)  \nGary M. Weiss Fordham University  [gaweiss@fordham.edu](gaweiss@fordham.edu)  \nAbstract  \nLetters of Recommendation (LOR) are key components of the undergraduate and graduate admissions process. A fair and objective evaluation of these LORs is difficult due to diverse applicant-recommender relationships, a lack of standardized criteria, and limited resources for reviewing the LORs. In this paper, we describe three criteria, relevance, specificity, and positivity, for characterizing the quality of an LOR. Approximately 4,000 LORs written in support of students applying to either a Master’s in Computer Science or a Master’sin Data Science degree are manually rated using these criteria along with rating guidelines developed for this study. Predictive models utilizing natural language processing and machine learning are trained to predict these ratings directly from the LOR text. The work described in this paper can aid in objective and automatic assessment of LORs, or help the admissions committee selectively review the LORs when resources are limited. This work can be extended to support the admissions process for other graduate and undergraduate programs.  \nKeywords: Graduate Admissions, Letters of Recommendation, Machine Learning, Natural Language Processing  \n1. Introduction  \nLetters of Recommendation (LOR) are used for both undergraduate and graduate admissions and can provide information about an applicant that may not be found in the other application materials. However, fair evaluation of these LORs is difficult due to their highly subjective nature, diverse applicant-recommender relationships,  \na lack of standards in what should be included, no standard evaluation criteria, and limited resources for reviewing the LORs. If a single reviewer cannot read all LORs, or if the applications are not all available at once, then some form of summary record must be maintained and used for comparison. While some universities utilize a scoring system to summarize the LORs, the practice often offloads the LOR scoring process to less trained staff. Nevertheless, this process is time-consuming and inevitably subject to inter-rater variability. An automated system can address the cost and consistency issues.  \nIn this paper, we describe three criteria, or dimensions, for evaluating any letter of recommendation: relevance, specificity, and positivity. A set of raters use these criteria, along with guidelines provided to them, to manually rate approximately 4,000 LORs associated with either a Master’s in Computer Science or Master’s in Data Science program. Predictive models using natural language processing and machine learning methods are subsequently trained on the LOR text to predict the manual ratings as the ground-truth labels. This study makes the following key contributions:  \n• It describes a set of criteria for evaluating any letter of recommendation and provides guidelines for manually applying these criteria to actual LORs.  \n• The LOR evaluation process is automated using natural language processing and machine learning methods.  \n• The performance for predicting the relevance, specificity, and positivity ratings is analyzed, providing insight into the potential for automatically measuring each criterion.  \nURI: [https://hdl.handle.net/10125/106534](https://hdl.handle.net/10125/106534)[ ](https://hdl.handle.net/10125/106534)[978-0-9981331-7-1](978-0-9981331-7-1)  \n(CC BY-NC-ND 4 .","cbCaic57WfHj2Ley","https://ap.wps.com/l/cbCaic57WfHj2Ley","pdf",711194,1,9,"English","en",105,"# Abstract\n# Introduction\n# Related work\n# Criteria for evaluating LORs\n# LOR dataset\n# Methodology and predictive models\n# Results and analysis\n# Conclusions and practical applications","[{\"question\":\"What criteria are used to evaluate letters of recommendation in this paper?\",\"answer\":\"The paper evaluates letters using three criteria: relevance, specificity, and positivity. These dimensions are used to characterize LOR quality and to guide manual ratings.\"},{\"question\":\"How are the training labels for machine learning models obtained?\",\"answer\":\"Approximately 4,000 letters are manually rated using the three criteria and guidelines developed for the study. These manual ratings serve as ground-truth labels for training predictive models.\"},{\"question\":\"How can the proposed approach help admissions committees?\",\"answer\":\"The approach enables more objective and automatic assessment of LORs. It also helps committees selectively review letters when resources are limited, reducing cost and minimizing individual bias and inter-rater variability.\"}]","A Quantitative Machine Learning Approach to Evaluating Letters of Recommendation | PDF",1785814190,23,{"code":4,"msg":31,"data":32},"ok",{"site_id":24,"language":23,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":28},"a-quantitative-machine-learning-approach-to-evaluating-letters-of-recommendation","",{"@graph":36,"@context":85},[37,54,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,51],{"item":41,"name":42,"@type":43,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":50},"https://docshare.wps.com/document/research-report/",3,{"item":52,"name":13,"@type":43,"position":53},"https://docshare.wps.com/document/a-quantitative-machine-learning-approach-to-evaluating-letters-of-recommendation/123017/",4,{"url":52,"name":13,"@type":55,"author":56,"headline":13,"publisher":58,"fileFormat":61,"inLanguage":23,"description":14,"dateModified":62,"datePublished":62,"encodingFormat":61,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":57},"Person",{"url":41,"name":59,"@type":60},"DocShare","Organization","application/pdf","2026-08-04",true,{"@type":65,"interactionType":66,"userInteractionCount":4},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What criteria are used to evaluate letters of recommendation in this paper?","Question",{"text":75,"@type":76},"The paper evaluates letters using three criteria: relevance, specificity, and positivity. These dimensions are used to characterize LOR quality and to guide manual ratings.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"How are the training labels for machine learning models obtained?",{"text":80,"@type":76},"Approximately 4,000 letters are manually rated using the three criteria and guidelines developed for the study. These manual ratings serve as ground-truth labels for training predictive models.",{"name":82,"@type":73,"acceptedAnswer":83},"How can the proposed approach help admissions committees?",{"text":84,"@type":76},"The approach enables more objective and automatic assessment of LORs. It also helps committees selectively review letters when resources are limited, reducing cost and minimizing individual bias and inter-rater variability.","https://schema.org",{"og:url":52,"og:type":87,"og:title":13,"og:site_name":59,"og:description":14},"article",{"robots":89,"canonical":52},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,127,130,134],{"id":20,"doc_module":4,"doc_module_name":46,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":53,"doc_module":4,"doc_module_name":46,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":46,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":46,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":46,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":21,"doc_module":4,"doc_module_name":46,"category_name":124,"show_sort_weight":125,"slug":126},"Religion & Spirituality",20,"religion-spirituality",{"id":125,"doc_module":4,"doc_module_name":46,"category_name":128,"show_sort_weight":125,"slug":129},"World Cup","world-cup",{"id":131,"doc_module":4,"doc_module_name":46,"category_name":132,"show_sort_weight":131,"slug":133},10,"Lifestyle","lifestyle",{"id":135,"doc_module":4,"doc_module_name":46,"category_name":136,"show_sort_weight":106,"slug":137},19,"General","general"]