[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-83248-en":3,"doc-seo-83248-105":30,"detail-sidebar-cat-0-en-105":90},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":21,"is_downloadable":21,"audit_status":21,"page_count":22,"language":23,"language_code":24,"site_id":25,"html_lang":24,"table_of_contents":26,"faqs":27,"seo_title":13,"seo_description":14,"update_tm":28,"read_time":29},83248,962075114101,"Seraphina","https://ap-avatar.wpscdn.com/avatar/e000253a75eb197efd?x-image-process=image/resize,m_fixed,w_180,h_180&k=1780044092746381165",8,"Research & Report","The Poisoned Chalice of LLM Evaluation Report","Large language models increasingly support software engineering, yet evaluation validity is undermined when benchmark instances were potentially seen during pretraining. Such data contamination can inflate reported performance and produce misleading conclusions about model capability. Because modern models disclose only partial training corpora, direct decontamination is often infeasible. This work introduces a competition that detects prior exposure via white-box membership inference on source code.","The Poisoned Chalice of LLM Evaluation Report  \nJonathan Katzy∗  \n[J.B.Katzy@TUDelft.nl](J.B.Katzy@TUDelft.nl)[ ](J.B.Katzy@TUDelft.nl)Delft University of Technology Delft, The Netherlands  \nAli Al-Kaswan  \n[A.Al-Kaswan@TUDelft.nl](A.Al-Kaswan@TUDelft.nl)[ ](A.Al-Kaswan@TUDelft.nl)Delft University of Technology Delft, The Netherlands  \nRazvan Mihai Popescu  \n[R.M.Popescu@TUDelft.nl](R.M.Popescu@TUDelft.nl)[ ](R.M.Popescu@TUDelft.nl)Delft University of Technology Delft, The Netherlands  \nZhou Yang  \n[ZY25@UAlberta.ca](ZY25@UAlberta.ca)[ ](ZY25@UAlberta.ca)University of Alberta, Alberta Machine Intelligence Institute Alberta, Canada  \narXiv :2607 .0748 1v 1 [ cs . SE] 8 Jul 2026  \nAbstract  \nLarge language models are increasingly used to evaluate and support software engineering tasks, yet the validity of these evaluations is often undermined by uncertainty about whether benchmark instances were seen during pretraining. This can lead to data contamination, which may inflate performance and result in misleading conclusions about model capability. Despite this, the training corpora of many modern models are only partially disclosed, making direct decontamination infeasible. This creates a need for practical methods that can detect a large language models’ prior exposure to training data without access to the full training corpus.  \nTo address this challenge, we organize the first Poisoned Chalice of LLM Evaluation Competition, co-located with the FSE-AIWare 2026 Competition Track. The competition frames contamination detection as a white-box membership inference task on source code and provides participants with curated datasets, target models, baseline attacks, and a final evaluation on a held-out model and dataset. This design encourages methods that generalize beyond superficial dataset artifacts and beyond a single training setting.  \nThis paper reports the setup and results of the competition. More broadly, the competition aims to catalyze the community around trustworthy LLM evaluation for software engineering.  \nCCS Concepts  \n• Computing methodologies → Natural language processing.  \nKeywords  \nLarge Language Models, Software Engineering, Privacy, Licensing  \nACM Reference Format:  \nJonathan Katzy, Ali Al-Kaswan, Razvan Mihai Popescu, and Zhou Yang.  \n2026. The Poisoned Chalice of LLM Evaluation Report. In 34thACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (FSE Companion ’26), July 05–09, 2026, Montreal, QC, Canada. ACM, New York, NY, USA, 4 pages. [https://doi.org/10.1145/](https://doi.org/10.1145/)[ ](https://doi.org/10.1145/)3803437.3807733  \n∗ Main contact  \nThis work is licensed under a Creative Commons Attribution 4 .0 International License. FSE Companion ’26, Montreal, QC, Canada  \n© 2026 Copyright held by the owner/author(s) .  \nACM ISBN 979-8-4007-2636-1/2026/07  \n[https://doi.org/10.1145/3803437.3807733](https://doi.org/10.1145/3803437.3807733)  \n1 Introduction  \nLarge Language Models (LLMs) have become central to modern software engineering research and practice. They have been applied to a wide range of tasks, including code generation, code completion, code summarization, test generation, vulnerability detection, and program repair [9, 15] . Their increasing effectiveness has also led to widespread adoption in developer tooling, where they are used to accelerate programming and support day-to-day software development activities.  \nAt the same time, the rapid progress of code-capable LLMs has made rigorous evaluation increasingly difficult. A central challenge is that LLMs are known to memorize portions of their training data and, in some cases, reproduce them verbatim or nearverbatim [3, 7, 16] . As a result, strong performance on a benchmark may not necessarily reflect robust generalization; instead, it may partially arise from prior exposure to benchmark instances during training. This issue, commonly referred to as data contamination, threatens th","cbCaidPuvymFwMeF","https://ap.wps.com/l/cbCaidPuvymFwMeF","pdf",592001,3,1,4,"English","en",105,"# Abstract\n# Introduction\n## Data contamination and evaluation validity\n## Decontamination limitations\n## Membership inference as an alternative\n# Competition setup and results","[{\"question\":\"为什么 LLM 在软件工程评测中的有效性会受到挑战？\",\"answer\":\"当评测基准样本在模型预训练阶段可能已被见过时，会导致数据污染，从而让评测结果无法真实反映泛化能力。\"},{\"question\":\"为什么直接去污染（decontamination）在现代模型上往往不可行？\",\"answer\":\"许多模型的训练语料并未完全公开，导致无法与完整训练集进行直接比对或去重。\"},{\"question\":\"文中如何处理数据污染检测这一问题？\",\"answer\":\"通过组织“Poisoned Chalice of LLM Evaluation”竞赛，将污染检测形式化为基于源代码的白盒成员推断任务，并提供定制数据集、目标模型与基线攻击方案。\"}]",1784186237,10,{"code":4,"msg":31,"data":32},"ok",{"site_id":25,"language":24,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":85,"head_meta":87,"extra_data":89,"updated_unix":28},"the-poisoned-chalice-of-llm-evaluation-report","",{"@graph":36,"@context":84},[37,52,67],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,50],{"item":41,"name":42,"@type":43,"position":21},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":20},"https://docshare.wps.com/document/research-report/",{"item":51,"name":13,"@type":43,"position":22},"https://docshare.wps.com/document/the-poisoned-chalice-of-llm-evaluation-report/83248/",{"url":51,"name":13,"@type":53,"author":54,"headline":13,"publisher":56,"fileFormat":59,"inLanguage":24,"description":14,"dateModified":60,"datePublished":61,"encodingFormat":59,"isAccessibleForFree":62,"interactionStatistic":63},"DigitalDocument",{"name":9,"@type":55},"Person",{"url":41,"name":57,"@type":58},"DocShare","Organization","application/pdf","2026-07-23","2026-07-16",true,{"@type":64,"interactionType":65,"userInteractionCount":20},"InteractionCounter",{"@type":66},"ViewAction",{"@type":68,"mainEntity":69},"FAQPage",[70,76,80],{"name":71,"@type":72,"acceptedAnswer":73},"为什么 LLM 在软件工程评测中的有效性会受到挑战？","Question",{"text":74,"@type":75},"当评测基准样本在模型预训练阶段可能已被见过时，会导致数据污染，从而让评测结果无法真实反映泛化能力。","Answer",{"name":77,"@type":72,"acceptedAnswer":78},"为什么直接去污染（decontamination）在现代模型上往往不可行？",{"text":79,"@type":75},"许多模型的训练语料并未完全公开，导致无法与完整训练集进行直接比对或去重。",{"name":81,"@type":72,"acceptedAnswer":82},"文中如何处理数据污染检测这一问题？",{"text":83,"@type":75},"通过组织“Poisoned Chalice of LLM Evaluation”竞赛，将污染检测形式化为基于源代码的白盒成员推断任务，并提供定制数据集、目标模型与基线攻击方案。","https://schema.org",{"og:url":51,"og:type":86,"og:title":13,"og:site_name":57,"og:description":14},"article",{"robots":88,"canonical":51},"index,follow",{"doc_id":7,"site_id":25},{"code":4,"msg":5,"data":91},[92,96,100,104,109,114,119,122,127,130,133],{"id":21,"doc_module":4,"doc_module_name":46,"category_name":93,"show_sort_weight":94,"slug":95},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":97,"show_sort_weight":98,"slug":99},"Literature",80,"literature",{"id":22,"doc_module":4,"doc_module_name":46,"category_name":101,"show_sort_weight":102,"slug":103},"Exam",70,"exam",{"id":105,"doc_module":4,"doc_module_name":46,"category_name":106,"show_sort_weight":107,"slug":108},5,"Comic",60,"comic",{"id":110,"doc_module":4,"doc_module_name":46,"category_name":111,"show_sort_weight":112,"slug":113},6,"Technology",50,"technology",{"id":115,"doc_module":4,"doc_module_name":46,"category_name":116,"show_sort_weight":117,"slug":118},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":120,"slug":121},30,"research-report",{"id":123,"doc_module":4,"doc_module_name":46,"category_name":124,"show_sort_weight":125,"slug":126},9,"Religion & Spirituality",20,"religion-spirituality",{"id":125,"doc_module":4,"doc_module_name":46,"category_name":128,"show_sort_weight":125,"slug":129},"World Cup","world-cup",{"id":29,"doc_module":4,"doc_module_name":46,"category_name":131,"show_sort_weight":29,"slug":132},"Lifestyle","lifestyle",{"id":134,"doc_module":4,"doc_module_name":46,"category_name":135,"show_sort_weight":105,"slug":136},19,"General","general"]