[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-119051-en":3,"doc-seo-119051-105":30,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":4,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":27,"seo_description":14,"update_tm":28,"read_time":29},119051,4398048950312,"Violet","https://ap-avatar.wpscdn.com/avatar/400002538284de19e3c?_k=1778320343897328908",8,"Research & Report","Diversified Ensembling - An Experiment in Crowdsourced Machine Learning","Crowdsourced machine learning on platforms like Kaggle enables many teams to iteratively improve predictive models, often culminating in ensembling for stronger final performance. This work expands a fair ML crowdsourcing framework where participants propose subgroup-specific improvements, integrating community feedback when subgroup unfairness is identifiable. The study conducts a medium-scale experiment with 46 teams predicting income from American Community Survey data, analyzes team strategies and the proposed system architecture, and provides practical guidance for deployment.","arXiv :2402 . 10795v1 [ cs .LG] 16 Feb 2024  \nDiversified Ensembling: An Experiment in Crowdsourced Machine Learning  \nIra Globus-Harris∗1, Declan Harrison†1, Michael Kearns2 , Pietro Perona2 , and Aaron Roth2  \n1 University of Pennsylvania  \n2 AWS AI Labs  \nFebruary 19, 2024  \nAbstract  \nCrowdsourced machine learning on competition platforms such as Kaggle is a popular and often effective method for generating accurate models. Typically, teams vie for the most accurate model, as measured by overall error on a holdout set, and it is common towards the end of such competitions for teams at the top of the leaderboard to ensemble or average their models outside the platform mechanism to get the final, best global model. In Globus-Harris et al. (2022), the authors developed an alternative crowdsourcing framework in the context of fair machine learning, in order to integrate community feedback into models when subgroup unfairness is present and identifiable. There, unlike in classical crowdsourced ML, participants deliberately specialize their efforts by working on subproblems, such as demographic subgroups in the service of fairness. Here, we take a broader perspective on this work: we note that within this framework, participants may both specialize in the service of fairness and simply to cater to their particular expertise (e.g., focusing on identifying bird species in an image classification task) . Unlike traditional crowdsourcing, this allows for the diversification of participants’ efforts and may provide a participation mechanism to a larger range of individuals (e.g. a machine learning novice who has insight into a specific fairness concern) . We present the first medium-scale experimental evaluation of this framework, with 46 participating teams attempting to generate models to predict income from American Community Survey data. We provide an empirical analysis of teams’ approaches, and discuss the novel system architecture we developed. From here, we give concrete guidance for how best to deploy such a framework.  \n1 Introduction  \nCompetition platforms are a popular framework for generating accurate machine learning models through communal efforts. Kaggle is the most popular of these “crowdsourced” machine learning platforms, boasting fifteen million user accounts and thousands of competitions to date.1 Companies and non-profits use the platform to publicly host competitions for learning tasks, often with rewards for the team with the highest performing model. One benefit of crowdsourcing models is that it gives a wider community access to the model development process: Kaggle, for instance, has been considered a mechanism for the “democratization”of data science to a broader audience, particularly in the context of crowdsourced models for tasks with societal utility Chou et al. (2014) . However, due to the standard structure of these competition frameworks, they do not truly leverage the expertise of all the competitors, and fail to explicitly align improvements  \n∗ Work done during an internship at Amazon †Work done during an internship at Amazon  \n1 [https://www.kaggle.com/](https://www.kaggle.com/)  \nin model fairness with competition success. Here, we implement and provide an empirical analysis of an alternate framework which provides such mechanisms.  \nIn Globus-Harris et al. (2022), the authors provide an alternative algorithmic framework for crowdsourcing machine learning models, which we implement here. Their framework was specifically designed for contexts where unfairness, in the form of disparate accuracy of models across identifiable subgroups of the distribution, is of concern, and where a model would be considered “fair” if the model’s error on each group is close to the Bayes optimal error on that group.2 In this framework, competitors compete against a global model f. At each round, they submit a function defining a group g and a model h which they claim has improved error compared to f when restricted","cbCaiok5U19u2WW1","https://ap.wps.com/l/cbCaiok5U19u2WW1","pdf",797133,1,23,"English","en",105,"# Introduction\n## Crowdsourced competition platforms and their limitations\n## Fair ML crowdsourcing framework and ensembling mechanism\n## Paper objective and experimental evaluation","[{\"question\":\"What is the core idea behind diversified ensembling in crowdsourced ML?\",\"answer\":\"Teams propose subgroup-specific model updates that are incorporated through a natural ensembling procedure, so the overall model improves while allowing participant diversification. The approach supports both fairness-driven specialization and expertise-driven contribution.\"},{\"question\":\"How does the proposed framework differ from standard Kaggle-style competitions?\",\"answer\":\"Standard competitions primarily reflect which team achieves the best overall holdout score, without explicitly aligning fairness improvements or enabling structured specialization. The proposed framework verifies subgroup-restricted improvements and integrates them into the global model across rounds.\"},{\"question\":\"What does the experimental evaluation in the paper involve?\",\"answer\":\"The authors run a medium-scale experiment with 46 participating teams to predict income using American Community Survey data. The work includes analysis of team approaches and discussion of the system architecture for deployment guidance.\"}]","Diversified Ensembling - An Experiment in Crowdsourced Machine Learning | PDF",1785722099,58,{"code":4,"msg":31,"data":32},"ok",{"site_id":24,"language":23,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":28},"diversified-ensembling-an-experiment-in-crowdsourced-machine-learning","",{"@graph":36,"@context":85},[37,54,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,51],{"item":41,"name":42,"@type":43,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":50},"https://docshare.wps.com/document/research-report/",3,{"item":52,"name":13,"@type":43,"position":53},"https://docshare.wps.com/document/diversified-ensembling-an-experiment-in-crowdsourced-machine-learning/119051/",4,{"url":52,"name":13,"@type":55,"author":56,"headline":13,"publisher":58,"fileFormat":61,"inLanguage":23,"description":14,"dateModified":62,"datePublished":62,"encodingFormat":61,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":57},"Person",{"url":41,"name":59,"@type":60},"DocShare","Organization","application/pdf","2026-08-03",true,{"@type":65,"interactionType":66,"userInteractionCount":4},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What is the core idea behind diversified ensembling in crowdsourced ML?","Question",{"text":75,"@type":76},"Teams propose subgroup-specific model updates that are incorporated through a natural ensembling procedure, so the overall model improves while allowing participant diversification. The approach supports both fairness-driven specialization and expertise-driven contribution.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"How does the proposed framework differ from standard Kaggle-style competitions?",{"text":80,"@type":76},"Standard competitions primarily reflect which team achieves the best overall holdout score, without explicitly aligning fairness improvements or enabling structured specialization. The proposed framework verifies subgroup-restricted improvements and integrates them into the global model across rounds.",{"name":82,"@type":73,"acceptedAnswer":83},"What does the experimental evaluation in the paper involve?",{"text":84,"@type":76},"The authors run a medium-scale experiment with 46 participating teams to predict income using American Community Survey data. The work includes analysis of team approaches and discussion of the system architecture for deployment guidance.","https://schema.org",{"og:url":52,"og:type":87,"og:title":13,"og:site_name":59,"og:description":14},"article",{"robots":89,"canonical":52},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":46,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":53,"doc_module":4,"doc_module_name":46,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":46,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":46,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":46,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":46,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":46,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":46,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":46,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]