[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-85558-en":3,"doc-seo-85558-105":29,"detail-sidebar-cat-0-en-105":90},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":4,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},85558,16904993612988,"Olivia Brown","https://ap-avatar.wpscdn.com/davatar_a8503ba1806abce46bf441b54a3ca4cd",8,"Research & Report","How Annotation Trains Annotators: Competence Development in Social Influence Recognition","Human data annotation is often treated as an objective reference, yet many tasks are inherently subjective and annotators’ judgments can change during the workflow. This study examines competence changes during social influence recognition. Twenty-five annotators from five groups—including experts and non-experts—annotated 1,021 dialogues using 20 social influence techniques. A 150-text subset was labeled twice to compare before/after effects. Mixed qualitative and quantitative methods, interviews, self-assessments, and LLM training show increased self-perceived competence and confidence, with improved data quality and stronger effects for expert groups.","arXiv :2604 .0295 1v2 [ cs .CL] 13 Jul 2026  \nHow Annotation Trains Annotators: Competence Development in Social Influence Recognition  \nMaciej Markiewicz 1[0009−0004−2882−6741], Beata Bajcar 1[0000−0001−5044−4070], Wiktoria Mieleszczenko-Kowszewicz 1[0000−0002−3948−268X], Aleksander  \nSzczęsny 1[0009−0003−6808−2321], Tomasz Adamczyk 1[0009−0005−9703−4630], Grzegorz Chodak 1[0000−0002−9604−482X], Karolina Ostrowska2[0009−0004−9959−2487], Aleksandra Sawczuk2[0009−0002−0677−7905], Jolanta Babiak 1[0000−0002−6604−7763], Jagoda Szklarczyk3[0009−0004−6784−8273], and Przemysław Kazienko 1[0000−0001−5868−356X]  \n1 Wrocław University of Science and Technology, Wrocław, Poland  \n{[name.surname}@pwr.edu.pl](name.surname}@pwr.edu.pl)  \n2 University of Silesia in Katowice, Katowice, Poland  \n3 SWPS University, Wrocław, Poland  \nAbstract. Human data annotation, especially when involving experts, is often treated as an objective reference. However, many annotation tasks are inherently subjective, and annotators’ judgments may evolve over time. This study investigates changes in the quality of annotators’work from a competence perspective during a process of social influence recognition. The study involved 25 annotators from five different groups, including both experts and non-experts, who annotated a dataset of 1,021 dialogues with 20 social influence techniques, along with intentions, reactions, and consequences. An initial subset of 150 texts was annotated twice – before and after the main annotation process – to enable comparison. To measure competence shifts, we combined qualitative and quantitative analyses of the annotated data, semi-structured interviews with annotators, self-assessment surveys, and Large Language Model training and evaluation on the comparison dataset. The results indicate a significant increase in annotators’ self-perceived competence and confidence.  \nMoreover, observed changes in data quality suggest that the annotation process may enhance annotator competence and that this effect is more pronounced in expert groups. The observed shifts in annotator competence have a visible impact on the performance of LLMs trained on their annotated data.  \nKeywords: annotation · learning process · LLMs · social influence · competence development  \nPreprint: Accepted to AIED 2026: The 27th International Conference on Artificial Intelligence in Education. This preprint has not undergone peer review or any post-submission improvements or corrections. The Version of Record of this contribution is published in LNCS vol 16584, and is available online at [https: // doi. org/ 10. 1007/ 978-3-032-29763-1_ 36](https: // doi. org/ 10. 1007/ 978-3-032-29763-1_ 36 . Please)[ . Please](https: // doi. org/ 10. 1007/ 978-3-032-29763-1_ 36 . Please) cite the published version.  \n2 M. Markiewicz et al.  \n1 Introduction  \nHuman annotation is foundational for creating high quality datasets and has been extensively studied. Expert annotation is often treated as objective ground truth, and aggregating multiple annotators typically yields satisfactory results despite individual errors. However, in subjective or partially-subjective tasks, such as the recognition of social influence, or when annotators are not experts, even human annotation may not be accurate or may change over the course of the process [7] . The aim of this study is to identify the changes in annotators’work and analyze their origins; whether it is the effect of increased competence, random noise, or a varying worldview. We then want to assess the impact of these changes on data quality and AI model training. The study is conducted in conjunction with our unpublished social influence recognition dataset but focuses solely on the annotation process. We formulate the following research questions:  \nRQ1 How does the competence of annotators regarding social influence recognition change over the course of the annotation process?  \nRQ2 How does the quality of the annotators’ work chang","cbCaivDaAdNkZCXC","https://ap.wps.com/l/cbCaivDaAdNkZCXC","pdf",554119,1,15,"English","en",105,"# Introduction\n# Related Work\n# The annotation process","[{\"question\":\"What is the main goal of the study on annotators?\",\"answer\":\"To identify how annotators’ competence and the quality of their work change during the annotation process, and to analyze the underlying causes of these shifts.\"},{\"question\":\"How did the study measure competence changes over time?\",\"answer\":\"It combined qualitative and quantitative analyses of the annotated data, semi-structured interviews, self-assessment surveys, and Large Language Model training and evaluation on the comparison subset.\"},{\"question\":\"What impact did competence shifts have on AI model training results?\",\"answer\":\"Observed changes in data quality indicate that the annotation process can enhance annotator competence, and this effect becomes more pronounced in expert groups, leading to visible changes in LLM performance when trained on their annotated data.\"}]",1784204541,38,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":85,"head_meta":87,"extra_data":89,"updated_unix":27},"how-annotation-trains-annotators-competence-development-in-social-influence-recognition","",{"@graph":35,"@context":84},[36,53,67],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/how-annotation-trains-annotators-competence-development-in-social-influence-recognition/85558/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":61,"encodingFormat":60,"isAccessibleForFree":62,"interactionStatistic":63},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-16",true,{"@type":64,"interactionType":65,"userInteractionCount":4},"InteractionCounter",{"@type":66},"ViewAction",{"@type":68,"mainEntity":69},"FAQPage",[70,76,80],{"name":71,"@type":72,"acceptedAnswer":73},"What is the main goal of the study on annotators?","Question",{"text":74,"@type":75},"To identify how annotators’ competence and the quality of their work change during the annotation process, and to analyze the underlying causes of these shifts.","Answer",{"name":77,"@type":72,"acceptedAnswer":78},"How did the study measure competence changes over time?",{"text":79,"@type":75},"It combined qualitative and quantitative analyses of the annotated data, semi-structured interviews, self-assessment surveys, and Large Language Model training and evaluation on the comparison subset.",{"name":81,"@type":72,"acceptedAnswer":82},"What impact did competence shifts have on AI model training results?",{"text":83,"@type":75},"Observed changes in data quality indicate that the annotation process can enhance annotator competence, and this effect becomes more pronounced in expert groups, leading to visible changes in LLM performance when trained on their annotated data.","https://schema.org",{"og:url":51,"og:type":86,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":88,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":91},[92,96,100,104,109,114,119,122,127,130,134],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":93,"show_sort_weight":94,"slug":95},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":97,"show_sort_weight":98,"slug":99},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":101,"show_sort_weight":102,"slug":103},"Exam",70,"exam",{"id":105,"doc_module":4,"doc_module_name":45,"category_name":106,"show_sort_weight":107,"slug":108},5,"Comic",60,"comic",{"id":110,"doc_module":4,"doc_module_name":45,"category_name":111,"show_sort_weight":112,"slug":113},6,"Technology",50,"technology",{"id":115,"doc_module":4,"doc_module_name":45,"category_name":116,"show_sort_weight":117,"slug":118},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":120,"slug":121},30,"research-report",{"id":123,"doc_module":4,"doc_module_name":45,"category_name":124,"show_sort_weight":125,"slug":126},9,"Religion & Spirituality",20,"religion-spirituality",{"id":125,"doc_module":4,"doc_module_name":45,"category_name":128,"show_sort_weight":125,"slug":129},"World Cup","world-cup",{"id":131,"doc_module":4,"doc_module_name":45,"category_name":132,"show_sort_weight":131,"slug":133},10,"Lifestyle","lifestyle",{"id":135,"doc_module":4,"doc_module_name":45,"category_name":136,"show_sort_weight":105,"slug":137},19,"General","general"]