[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-85177-en":3,"doc-seo-85177-105":29,"detail-sidebar-cat-0-en-105":90},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":4,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},85177,962075114765,"Quinn","https://ap-avatar.wpscdn.com/davatar_a8503ba1806abce46bf441b54a3ca4cd",8,"Research & Report","When Data Imbalance Helps: Robust Generalization Through Shortcut Saturation","Study of robust generalization under spurious correlations analyzes tasks where a shortcut feature aligns with the true label during training yet reverses on an adversarial held-out split. Varying the spurious ratio r and model capacity, the work reports a counterintuitive effect: data imbalance can increase robust accuracy for sufficiently capable models. On a Max-Parity-Sum-Parity synthetic task, two-layer two-head transformers reach 100% adversarial accuracy in 77% of seeds at r=0.90, despite 0% at r=0.50, while one-layer models reverse. Mechanistic analyses link gradient-conflict dynamics and circuit evolution to shortcut saturation.","When Data Imbalance Helps: Robust Generalization Through Shortcut Saturation  \nCheng-Ting Chou∗  \nUniversity of California, Los Angeles [ctchou3@cs.ucla.edu](ctchou3@cs.ucla.edu)  \nDuc Binh Hoang∗  \nPurdue University [hoang112@purdue.edu](hoang112@purdue.edu)  \narXiv :2607 . 10 1 16v 1 [ cs .LG] 11 Jul 2026  \nAbstract  \nWe study robust generalization under spurious correlations: tasks where a shortcut feature is correlated with the true label in training but anti-correlated in an adversarial held-out split. Varying the spurious ratio r (the fraction of training examples where shortcut = true label) and model capacity, we find a counterintuitive result: data imbalance promotes generalization in sufficiently capable models. On a synthetic task where the true label is sum parity of an integer sequence and the shortcut is the parity of the maximum-valued element, a 2-layer, 2-head transformer generalized (reached 100% adversarial accuracy) in 0% of seeds at r=0 .50 but 77% of seeds at r=0 .90. The effect is absent in 1-layer models, where imbalance instead traps the model on the shortcut. Through mechanistic analysis—gradient conflict dynamics, circuit evolution, and QK/OV circuit ablations—we characterize a mechanistic pathway consistent with imbalance promoting generalization.  \n1 Introduction  \nSpurious correlations are one of the central failure modes of learned models [Geirhos et al., 2020, Shah et al., 2020] . When a feature is highly predictive in training but unreliable at test time, models learn to exploit it as a shortcut, and the standard prescription is to correct for this by balancing the dataset—equalizing shortcut-consistent and anti-shortcut examples so neither dominates gradient descent [Sagawa et al., 2020] . We challenge this prescription.  \nIn a controlled synthetic setting where models must predict sum parity of an integer sequence but can exploit the parity of the maximum-valued element as a shortcut (the Max-Parity-Sum-Parity task, defined in 3), we find that increasing the spurious ratio r—the fraction of training examples where shortcut equals true label—from 0.50 to 0.90 substantially increases the probability of robust generalization, but only in models with sufficient capacity. Two-layer, two-head transformers generalize in 0% of seeds at r=0 .50 and 77% of seeds at r=0 .90; single-layer models show the opposite trend, with higher imbalance trapping them on the shortcut.  \nWe hypothesize that the mechanism underlying this reversal involves shortcut saturation—the regime in which shortcut-consistent examples achieve near-zero loss, causing their gradients to vanish. When imbalance is high, the shortcut circuit rapidly achieves near-perfect accuracy on the majority of training examples, bringing them into this saturated regime. The anti-shortcut minority, consistently misclassified, continues to produce large gradients—amplified by a factor of roughly r/(1−r), reaching 9:1 at r=0 .90 [You et al., 2025] . In capable models, this amplified adversarial gradient appears to support a structural reorganization of the attention circuit. At balanced ratios, no such saturation appears to occur: the two gradient sources remain persistently opposed and neither side gains sufficient gradient momentum. These results suggest that shortcut saturation may function as a precondition for generalization rather than an obstacle to it.  \n∗Equal contribution.  \nPreprint.  \nOur contributions are:  \n1. Imbalance × capacity interaction. Increasing the spurious ratio from 0.5 to 0.9 raises generalization rate from 0% to 77% in two-layer transformers while reducing it in one-layer models, establishing a capacity threshold between one and two transformer layers.  \n2. Mechanistic pathway. Gradient conflict analysis, circuit evolution tracking, and QK/OVablations characterize a pathway consistent with shortcut saturation amplifying adversarial gradients and supporting structural circuit reorganization in capable models.  \n3. Unifying ","cbCainhOy2yaq52Z","https://ap.wps.com/l/cbCainhOy2yaq52Z","pdf",420329,1,16,"English","en",105,"# Introduction\n# Background and Related Work","[{\"question\":\"What problem does the paper address?\",\"answer\":\"The paper studies when models achieve robust generalization despite spurious correlations, specifically when a shortcut feature is correlated with the true label in training but anti-correlated in an adversarial test split.\"},{\"question\":\"How does changing the spurious ratio r affect generalization?\",\"answer\":\"Increasing r from 0.50 to 0.90 substantially increases the probability of robust generalization for sufficiently capable models, while balanced ratios do not show the same effect.\"},{\"question\":\"Why does data imbalance help in some models but hurt in others?\",\"answer\":\"The authors attribute the reversal to shortcut saturation: in capable models, shortcut-consistent examples quickly reach near-zero loss and their gradients vanish, while anti-shortcut minority examples produce amplified adversarial gradients that support structural circuit reorganization. In one-layer models, imbalance instead traps the model on the shortcut.\"}]",1784201558,40,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":85,"head_meta":87,"extra_data":89,"updated_unix":27},"when-data-imbalance-helps-robust-generalization-through-shortcut-saturation","",{"@graph":35,"@context":84},[36,53,67],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/when-data-imbalance-helps-robust-generalization-through-shortcut-saturation/85177/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":61,"encodingFormat":60,"isAccessibleForFree":62,"interactionStatistic":63},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-16",true,{"@type":64,"interactionType":65,"userInteractionCount":4},"InteractionCounter",{"@type":66},"ViewAction",{"@type":68,"mainEntity":69},"FAQPage",[70,76,80],{"name":71,"@type":72,"acceptedAnswer":73},"What problem does the paper address?","Question",{"text":74,"@type":75},"The paper studies when models achieve robust generalization despite spurious correlations, specifically when a shortcut feature is correlated with the true label in training but anti-correlated in an adversarial test split.","Answer",{"name":77,"@type":72,"acceptedAnswer":78},"How does changing the spurious ratio r affect generalization?",{"text":79,"@type":75},"Increasing r from 0.50 to 0.90 substantially increases the probability of robust generalization for sufficiently capable models, while balanced ratios do not show the same effect.",{"name":81,"@type":72,"acceptedAnswer":82},"Why does data imbalance help in some models but hurt in others?",{"text":83,"@type":75},"The authors attribute the reversal to shortcut saturation: in capable models, shortcut-consistent examples quickly reach near-zero loss and their gradients vanish, while anti-shortcut minority examples produce amplified adversarial gradients that support structural circuit reorganization. In one-layer models, imbalance instead traps the model on the shortcut.","https://schema.org",{"og:url":51,"og:type":86,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":88,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":91},[92,96,100,104,109,114,118,121,126,129,133],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":93,"show_sort_weight":94,"slug":95},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":97,"show_sort_weight":98,"slug":99},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":101,"show_sort_weight":102,"slug":103},"Exam",70,"exam",{"id":105,"doc_module":4,"doc_module_name":45,"category_name":106,"show_sort_weight":107,"slug":108},5,"Comic",60,"comic",{"id":110,"doc_module":4,"doc_module_name":45,"category_name":111,"show_sort_weight":112,"slug":113},6,"Technology",50,"technology",{"id":115,"doc_module":4,"doc_module_name":45,"category_name":116,"show_sort_weight":28,"slug":117},7,"Healthcare","healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":119,"slug":120},30,"research-report",{"id":122,"doc_module":4,"doc_module_name":45,"category_name":123,"show_sort_weight":124,"slug":125},9,"Religion & Spirituality",20,"religion-spirituality",{"id":124,"doc_module":4,"doc_module_name":45,"category_name":127,"show_sort_weight":124,"slug":128},"World Cup","world-cup",{"id":130,"doc_module":4,"doc_module_name":45,"category_name":131,"show_sort_weight":130,"slug":132},10,"Lifestyle","lifestyle",{"id":134,"doc_module":4,"doc_module_name":45,"category_name":135,"show_sort_weight":105,"slug":136},19,"General","general"]