[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"detail-sidebar-cat-1-en-105":3,"doc-seo-237845-105":53,"doc-detail-237845-en":126},{"code":4,"msg":5,"data":6},0,"success",[7,14,19,24,29,34,39,44,49],{"id":8,"doc_module":9,"doc_module_name":10,"category_name":11,"show_sort_weight":12,"slug":13},11,1,"Template","Presentations",90,"presentations",{"id":15,"doc_module":9,"doc_module_name":10,"category_name":16,"show_sort_weight":17,"slug":18},12,"Resumes",80,"resumes",{"id":20,"doc_module":9,"doc_module_name":10,"category_name":21,"show_sort_weight":22,"slug":23},14,"Invoices",70,"invoices",{"id":25,"doc_module":9,"doc_module_name":10,"category_name":26,"show_sort_weight":27,"slug":28},15,"Posters",60,"posters",{"id":30,"doc_module":9,"doc_module_name":10,"category_name":31,"show_sort_weight":32,"slug":33},16,"Social Media",50,"social-media",{"id":35,"doc_module":9,"doc_module_name":10,"category_name":36,"show_sort_weight":37,"slug":38},17,"Forms",40,"forms",{"id":40,"doc_module":9,"doc_module_name":10,"category_name":41,"show_sort_weight":42,"slug":43},18,"Letters",30,"letters",{"id":45,"doc_module":9,"doc_module_name":10,"category_name":46,"show_sort_weight":47,"slug":48},21,"Paper Templates",5,"papers-templates",{"id":50,"doc_module":9,"doc_module_name":10,"category_name":51,"show_sort_weight":4,"slug":52},158,"General","general-158",{"code":4,"msg":54,"data":55},"ok",{"site_id":56,"language":57,"slug":58,"title":59,"keywords":60,"description":61,"schema_data":62,"social_meta":119,"head_meta":121,"extra_data":123,"updated_unix":125},105,"en","a-collaborative-mechanism-for-crowdsourcing-prediction-problems","A Collaborative Mechanism for Crowdsourcing Prediction Problems","","Machine Learning competitions like the Netflix Prize have enabled crowdsourcing for prediction tasks, yet their incentive design often discourages sharing and collaboration. This paper introduces a Crowdsourced Learning Mechanism where participants collaboratively “learn” by updating a published hypothesis through wagering. The mechanism publishes the current hypothesis, accepts wagers on proposed modifications, and rewards participants based on how much their update improves performance on a released test set, aligning incentives with measurable gains.",{"@graph":63,"@context":118},[64,80,101],{"@type":65,"itemListElement":66},"BreadcrumbList",[67,71,74,77],{"item":68,"name":69,"@type":70,"position":9},"https://docshare.wps.com","Home","ListItem",{"item":72,"name":10,"@type":70,"position":73},"https://docshare.wps.com/template/",2,{"item":75,"name":51,"@type":70,"position":76},"https://docshare.wps.com/template/general/",3,{"item":78,"name":59,"@type":70,"position":79},"https://docshare.wps.com/template/a-collaborative-mechanism-for-crowdsourcing-prediction-problems/237845/",4,{"url":78,"name":59,"@type":81,"image":82,"author":87,"headline":59,"publisher":90,"fileFormat":93,"inLanguage":57,"description":61,"dateModified":94,"datePublished":95,"encodingFormat":93,"isAccessibleForFree":96,"interactionStatistic":97},"DigitalDocument",{"url":83,"@type":84,"width":85,"height":86},"https://docshare.wps.com/thumbnails/a-collaborative-mechanism-for-crowdsourcing-prediction-problems/237845.png","ImageObject",442,249,{"name":88,"@type":89},"Liam","Person",{"url":68,"name":91,"@type":92},"DocShare","Organization","application/pdf","2026-09-26","2026-09-11",true,{"@type":98,"interactionType":99,"userInteractionCount":76},"InteractionCounter",{"@type":100},"ViewAction",{"@type":102,"mainEntity":103},"FAQPage",[104,110,114],{"name":105,"@type":106,"acceptedAnswer":107},"What weaknesses of existing crowdsourced prediction competitions does the paper identify?","Question",{"text":108,"@type":109},"The paper highlights anti-collaboration incentives, misaligned winner-take-all rewards and benchmark thresholds, and a structure that discourages proprietary or non-open methods because winners must reveal their algorithms.","Answer",{"name":111,"@type":106,"acceptedAnswer":112},"How does the proposed Crowdsourced Learning Mechanism work?",{"text":113,"@type":109},"It publishes a current hypothesis and lets participants place wagers on modifications to that hypothesis; the updated hypothesis is posted, and after a wagering period the test set is revealed to determine payouts.",{"name":115,"@type":106,"acceptedAnswer":116},"What determines a participant’s profit in the mechanism?",{"text":117,"@type":109},"A trader’s profit scales with how much their proposed modification improves performance on the released test data, directly tying incentives to measurable predictive gains.","https://schema.org",{"og:url":78,"og:type":120,"og:title":59,"og:site_name":91,"og:description":61},"article",{"robots":122,"canonical":78},"index,follow",{"doc_id":124,"site_id":56},237845,1789126739,{"code":4,"msg":5,"data":127},{"doc_id":124,"user_id":128,"nickname":88,"user_avatar":129,"doc_module":9,"category_id":50,"category_name":51,"doc_title":59,"doc_description":61,"doc_content":130,"file_id":131,"file_url":132,"file_type":133,"file_size":134,"view_count":76,"is_deleted":4,"is_public":9,"is_downloadable":9,"audit_status":9,"page_count":135,"language":136,"language_code":57,"site_id":56,"html_lang":57,"table_of_contents":137,"faqs":138,"seo_title":139,"seo_description":61,"update_tm":125,"read_time":76},8796095461564,"https://ap-avatar.wpscdn.com/davatar_155a257f0dc6eb9ab79c44ca47cae57d","A Collaborative Mechanism for Crowdsourcing Prediction Problems  \nJacob Abernethy  \nDivision of Computer Science University of California at Berkeley [jake@cs.berkeley.edu](jake@cs.berkeley.edu)  \nRafael M. Frongillo  \nDivision of Computer Science University of California at Berkeley [raf@cs.berkeley.edu](raf@cs.berkeley.edu)  \nAbstract  \nMachine Learning competitions such as the Netﬂix Prize have proven reasonably successful as a method of “crowdsourcing” prediction tasks. But these competitions have a number of weaknesses, particularly in the incentive structure they create for the participants. We propose a new approach, called a Crowdsourced Learning Mechanism, in which participants collaboratively “learn” a hypothesis for a given prediction task. The approach draws heavily from the concept of a prediction market, where traders bet on the likelihood of a future event. In our framework, the mechanism continues to publish the current hypothesis, and participants can modify this hypothesis by wagering on an update. The critical incentive property is that a participant will proﬁt an amount that scales according to how much her update improves performance on a released test set.  \n1 Introduction  \nThe last several years has revealed a new trend in Machine Learning: prediction and learning problems rolled into prize-driven competitions. One of the ﬁrst, and certainly the most well-known, was the Netﬂix prize released in the Fall of 2006 . Netﬂix, aiming to improve the algorithm used to predict users' preferences on its database of ﬁlms, released a dataset of 100M ratings to the public and asked competing teams to submit a list of predictions on a test set withheld from the public. Netﬂix offered $1,000,000 to the ﬁrst team achieving prediction accuracy exceeding a given threshold, agoal that was eventually met. This competitive model for solving a prediction task has been used for a range of similar competitions since, and there is even a new company (kaggle.com) that creates and hosts such competitions. Such prediction competitions have proven quite valuable fora couple of important reasons: (a) they leverage the abilities and knowledge of the public at large, commonly known as “crowdsourcing”, and (b) they provide an incentivized mechanism for an individual or team to apply their own knowledge and techniques which could be particularly beneﬁcialto the problem at hand. This type of prediction competition provides a nice tool for companies andinstitutions that need help with a given prediction task yet can not afford to hire an expert. The potential leverage can be quite high: the Netﬂix prize winners apparently spent more than $1,000,000 in effort on their algorithm alone.  \nDespite the extent of its popularity, is the Netﬂix competition model the ideal way to “crowdsource”a learning problem? We note several weaknesses:  \nIt is anti-collaborative. Competitors are strongly incentivized to keep their techniques private. This is in stark contrast to many other projects that rely on crowdsourcing – Wikipedia being a prime example, where participants must build off the work of others. Indeed, in the case of the Netﬂix prize, not only do leading participants lack incentives to share, but the work of non-winning competitors is effectively wasted.  \nThe incentives are skewed and misaligned. The winner-take-all prize structure means that second place is as good as having not competed at all. This ultimately leads to an equilibrium where only a few teams are actually competing, and where potential new teams never form since catching up seems so unlikely. In addition, the ﬁxed achievement benchmark, set by Netﬂix as a 10% improvement in prediction RMSE over a baseline, leads to misaligned incentives. Effectively, the prize structure implies that an improvement of %9.9 percent is worth nothing to Netﬂix, whereas a 20% improvement is still only worth $1,000,000 to Netﬂix. This is clearly not optimal.  \nThe nature of the competition preclude","cbCaigOdTZN2tTd0","https://ap.wps.com/l/cbCaigOdTZN2tTd0","pdf",289103,9,"English","# Abstract\n# Introduction","[{\"question\":\"What weaknesses of existing crowdsourced prediction competitions does the paper identify?\",\"answer\":\"The paper highlights anti-collaboration incentives, misaligned winner-take-all rewards and benchmark thresholds, and a structure that discourages proprietary or non-open methods because winners must reveal their algorithms.\"},{\"question\":\"How does the proposed Crowdsourced Learning Mechanism work?\",\"answer\":\"It publishes a current hypothesis and lets participants place wagers on modifications to that hypothesis; the updated hypothesis is posted, and after a wagering period the test set is revealed to determine payouts.\"},{\"question\":\"What determines a participant’s profit in the mechanism?\",\"answer\":\"A trader’s profit scales with how much their proposed modification improves performance on the released test data, directly tying incentives to measurable predictive gains.\"}]","A Collaborative Mechanism for Crowdsourcing Prediction Problems | PDF"]