[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-84231-en":3,"doc-seo-84231-105":29,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},84231,962075114101,"Seraphina","https://ap-avatar.wpscdn.com/avatar/e000253a75eb197efd?x-image-process=image/resize,m_fixed,w_180,h_180&k=1780044092746381165",8,"Research & Report","Should We Dangle a Carrot The Effect of Performance-based Incentives in Visualization Experiments","A core visualization research challenge is identifying which visual encodings best support specific user tasks. Controlled experiments establish relative effectiveness, yet experiment design often includes subjective, sometimes ad hoc choices such as training and performance-based incentives, whose effects on empirical conclusions are poorly understood. This work examines monetary rewards as a design factor using two crowdsourced studies that vary both visualization type and incentive presence across perceptual correlation tasks and decision-making from weather data.","Should We Dangle a Carrot? The Effect of Performance-based Incentives in Visualization Experiments  \nAbhraneel Sarma  , Matthew Kay  , Sheng Long  , Michael Correll  , Alexander Lex   \nAbstract—A perennial research question in visualization involves identifying which visual encodings for a particular dataset are most effective for users in performing a specific task. The relative effectiveness of the different encodings are commonly identified through controlled experiments. However, designing an experiment involves making many, often ad hoc, decisions about the experimental setup such as whether to include a training module, whether to provide performance-based incentives to participants, etc. Yet, thereis limited guidance on how these decisions should be made, and we do not fully understand the impact of these subjective decisionson empirical results. In this paper, we investigate the impact of one such key design decision—monetary rewards. Specifically, we ask: does providing or not providing participants with performance-based financial incentives affect the results and the conclusions that we draw from visualization studies? We conducted two crowdsourced studies investigating the impact of incentives on (i) a low-level, perceptual task (perception of correlations in scatterplots or parallel coordinate plots), and (ii) a task involving reasoning (decision-making based on a weather forecast represented as intervals or density plots) . In each of these studies, we manipulate both the visual representation and the presence of incentives as between-subject conditions. We expected to find no effect of incentiveson the perceptual task, but to see an effect for the decision-making task. However, we found no effect on task performance in either study. While these are results of only two studies and should be replicated, they suggest that performance-based financial incentives may not always have the intended effect on participants that we presumed, and calls for a reflection of how incentivized studies should be designed. A copy of this paper and all supplemental materials are available at [https://osf.io/t8eah](https://osf.io/t8eah).  \nIndex Terms—Graphical Perception, Decision Making, Quantitative Methods, Experimental Design, Incentives.  \narXiv :2607 .07463v 1 [ cs .HC] 8 Jul 2026  \n1 INTRODUCTION  \nWhen designing a visualization, one of the goals of a designer is to communicate the data effectively to the viewer. This typically means representing the data using visual encodings which are easy for viewers to decode. To identify the relative effectiveness of visual encodings for a particular task, visualization researchers turn to empirical studies where they compare how well participants perform on a task using two or more visual representations. The encoding which allows the average participant to perform better on a task is then considered more effective for that task. However, in the process of designing an experiment researchers need to make many, often subjective or ad hoc decisions that may have unknown effects. Here, we draw attention to the potential issue of researcher degrees of freedom in the empirical studies that visualization researchers use for determining the effectiveness of various visual encodings, and how this flexibility can impact our theoretical understanding.  \nThis work was partly motivated by our prior experience designing an experiment to study the effect of a new visualization on a decisionmaking task [56] . Initially, we provided participants with a description of how the data was encoded in the visualization and what the information meant; yet we observed a lot of variance in how participants interpreted the visualization and used the information to make decisions. We speculated that some of this variance could be attributable to people misunderstanding how to read the chart [44] . Subsequently, when we included more explicit information on how to read the chart during onboarding, we observed less heterog","cbCaim9s1406WPxT","https://ap.wps.com/l/cbCaim9s1406WPxT","pdf",981463,1,13,"English","en",105,"# Introduction\n## Research motivation and context\n## Research question and studies","[{\"question\":\"What question does the paper investigate about visualization experiments?\",\"answer\":\"Whether providing or withholding performance-based financial incentives changes study results and the conclusions drawn from visualization research.\"},{\"question\":\"How many studies and what tasks are used to test incentives?\",\"answer\":\"Two crowdsourced studies are conducted: one on a perceptual task (correlation perception in scatterplots/parallel coordinates) and one on a reasoning task (decision-making from weather information shown with interval or density plots).\"},{\"question\":\"What effect did the authors find for performance-based monetary incentives?\",\"answer\":\"No effect on task performance was observed in either study, suggesting incentives may not have the intended impact and motivating reconsideration of how incentive-based studies are designed.\"}]",1784194214,33,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":27},"should-we-dangle-a-carrot-the-effect-of-performance-based-incentives-in-visualization-experiments","",{"@graph":35,"@context":85},[36,53,68],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/should-we-dangle-a-carrot-the-effect-of-performance-based-incentives-in-visualization-experiments/84231/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-17","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What question does the paper investigate about visualization experiments?","Question",{"text":75,"@type":76},"Whether providing or withholding performance-based financial incentives changes study results and the conclusions drawn from visualization research.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"How many studies and what tasks are used to test incentives?",{"text":80,"@type":76},"Two crowdsourced studies are conducted: one on a perceptual task (correlation perception in scatterplots/parallel coordinates) and one on a reasoning task (decision-making from weather information shown with interval or density plots).",{"name":82,"@type":73,"acceptedAnswer":83},"What effect did the authors find for performance-based monetary incentives?",{"text":84,"@type":76},"No effect on task performance was observed in either study, suggesting incentives may not have the intended impact and motivating reconsideration of how incentive-based studies are designed.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":45,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":45,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":45,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":45,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":45,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":45,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":45,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]