[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-84761-en":3,"doc-seo-84761-105":29,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},84761,4398048950312,"Violet","https://ap-avatar.wpscdn.com/avatar/400002538284de19e3c?_k=1778320343897328908",8,"Research & Report","What You See Is What You Get: Observation-Aligned Supervision for Chart-to-Code Generation","Chart-to-code generation is often trained with supervised fine-tuning on reference plotting scripts, assuming the gold code is fully observable from the rendered image. This assumption breaks for many chart types because multiple underlying data arrays can yield visually identical outputs. Boxplots, pie charts, and histograms expose only aggregated or proportion/bin-level quantities. Observation-Aligned supervision rewrites latent raw-data targets into chart-identifiable targets, aligning training with observable semantics. Experiments on ChartMimic and ChartX using multiple VLMs show consistent improvements in recovering observable values, including under executable evaluation.","What You See Is What You Get: Observation-Aligned Supervision for  \nChart-to-Code Generation  \nTianhao Niu Qingfu Zhu Wanxiang Che  \nResearch Center for Social Computing and Interactive Robotics Harbin Institute of Technology, China  \narXiv :2607 .04726v 1 [ cs .CL] 6 Jul 2026  \nAbstract  \nChart-to-code generation is commonly trained with supervised fine-tuning on reference plotting scripts, implicitly treating the gold code as a fully observable target. We argue that this assumption is often invalid: many chart programs contain latent raw variables that cannot be uniquely recovered from the rendered image. For example, a boxplot exposes summary statistics rather than original samples, a pie chart reveals proportions rather than arbitrary raw values, and a histogram shows bin-level mass rather than individual observations. Supervising models to reproduce such non-identifiable quantities encourages hallucination and over-specified code generation. We introduce Observation-Aligned supervision, arewriting framework that replaces latent rawdata targets with quantities constrained by the visual observation: box statistics for boxplots, wedge percentages for pie charts, and bin weights for histograms. Applying this framework to chart-to-code training data from two sources, we obtain the Observation-Aligned supervision target data. Experiments across multiple VLMs on ChartMimic and ChartX demonstrate consistent improvements in observable value recovery, including under bothexecutable evaluation. Our results suggest that improving chart-to-code models requires not only more data or advanced learning objectives  \nor algorithms, but also supervision targets that respect what is identifiable from the chart image.  \n1 Introduction  \nChart-to-code generation (Wu et al., 2025 ; Yang et al., 2025) aims to recover executable plotting programs from chart images, providing a structured and reproducible representation of visualizations. Compared with textual chart descriptions, executable code can preserve fine-grained layout, style, and data-related information, and has therefore become an increasingly important target for  \nVLM   \nimport matplotlib.pyplot as plt. . .  \nLatent variables can ’t be observed from the image  \nweekly_yields = [  \n[4 .0 , 4 .2 , 3 .5 , 4 .1 , 3 .8 , 4 .4 , 4 .2 , 4 .0],  \n[3 .2 , 2 .8 , 3 .1 , 3 .3 , 2 .9 , 3 .5 , 3 .0 , 3 .2]]  \n. . .  \nbplot=axs[0] .boxplot(weekly_yields ,notch=True). . .  \ntotal_yields=[sum(w) for w in weekly_yields]  \nbars = axs [1] .bar(vegetables , total_yields , color=colors , edgecolor= 'black ' )  \n. . .  \nbplot = axs [0] .bxp(weekly_yields , shownotches=True ,patch_artist=True)  \n. . .  \ntotal_yields = [32 .2 , 25 .0]  \nbars = axs [1] .bar(vegetables , total_yields , color=colors , edgecolor= 'black ' )  \n. . .  \nRaw-Data Supervision Target  \nObservation-Aligned Supervision Target  \nFigure 1: Observation-aligned supervision for boxplot chart-to-code. Raw-Data supervision treats sample arrays passed to ax.boxplot as targets, although they are not uniquely identifiable from the rendered chart. We rewrite the target into precomputed box statistics rendered via ax.bxp and explicitly propagate derived quantities used by other chart elements. We refer to the result supervision as Observation-Aligned Supervision Target.  \nmultimodal chart understanding. (Shen et al., 2026 ; Zhao et al., 2025a) Recent work has made substantial progress by scaling chart-code datasets (Zhao et al., 2025a ; Niu et al., 2025 ; Tan et al., 2025) and designing stronger post-training objectives (Chenet al., 2026 ; Tan et al., 2025 ; Tang et al., 2026b ; He et al., 2026) for visually faithful reconstruction. However, most existing approaches still inherit a common assumption from supervised fine-tuning: the reference plotting script code is treated as a unique and fully observable target.  \nThis assumption is often false. A rendered chart is not an injective observation of the program that produced it. Many different progr","cbCaiowvjQFWi98E","https://ap.wps.com/l/cbCaiowvjQFWi98E","pdf",2485727,1,21,"English","en",105,"# Abstract\n# 1 Introduction","[{\"question\":\"Why does standard chart-to-code supervised fine-tuning lead to problems?\",\"answer\":\"It treats reference plotting code as a unique, fully observable target, but rendered charts are not injective observations of the underlying program. Latent variables in the code may be unidentifiable from the image, creating an ill-posed inverse problem.\"},{\"question\":\"What is “latent-observation mismatch” in chart-to-code supervision?\",\"answer\":\"It is a systematic mismatch where the reference code may be executable and visually valid, yet contains latent degrees of freedom that should not be treated as unique gold targets. This can reward memorization or hallucination rather than recovering observable semantics.\"},{\"question\":\"How does Observation-Aligned supervision rewrite training targets?\",\"answer\":\"It replaces non-identifiable latent quantities with quantities constrained by visual observations. Boxplots are rewritten to box statistics, pie charts to wedge percentages (or normalized proportions), and histograms to bin edges and bin weights, aligning targets with what the chart reveals.\"}]",1784198090,53,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":27},"what-you-see-is-what-you-get-observation-aligned-supervision-for-chart-to-code-generation","",{"@graph":35,"@context":85},[36,53,68],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/what-you-see-is-what-you-get-observation-aligned-supervision-for-chart-to-code-generation/84761/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-17","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"Why does standard chart-to-code supervised fine-tuning lead to problems?","Question",{"text":75,"@type":76},"It treats reference plotting code as a unique, fully observable target, but rendered charts are not injective observations of the underlying program. Latent variables in the code may be unidentifiable from the image, creating an ill-posed inverse problem.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"What is “latent-observation mismatch” in chart-to-code supervision?",{"text":80,"@type":76},"It is a systematic mismatch where the reference code may be executable and visually valid, yet contains latent degrees of freedom that should not be treated as unique gold targets. This can reward memorization or hallucination rather than recovering observable semantics.",{"name":82,"@type":73,"acceptedAnswer":83},"How does Observation-Aligned supervision rewrite training targets?",{"text":84,"@type":76},"It replaces non-identifiable latent quantities with quantities constrained by visual observations. Boxplots are rewritten to box statistics, pie charts to wedge percentages (or normalized proportions), and histograms to bin edges and bin weights, aligning targets with what the chart reveals.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":45,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":45,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":45,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":45,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":45,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":45,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":45,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]