[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-seo-269382-105":3,"detail-sidebar-cat-0-en-105":81,"doc-detail-269382-en":131},{"code":4,"msg":5,"data":6},0,"ok",{"site_id":7,"language":8,"slug":9,"title":10,"keywords":11,"description":12,"schema_data":13,"social_meta":74,"head_meta":76,"extra_data":78,"updated_unix":80},105,"en","clip-more-robust-to-long-tailed-pre-training-data-a-controlled-study-for-transferable-insights-neurips-2024","CLIP More Robust to Long-Tailed Pre-Training Data - A Controlled Study for Transferable Insights - NeurIPS 2024","","Severe data imbalance appears naturally in web-scale vision-language datasets, yet CLIP pre-trained on such data shows stronger robustness than supervised learning while learning representations that generalize across tasks. Controlled experiments identify a mechanism: CLIP’s pretext task becomes a dynamic classification setting where only a subset of classes is present, isolating dominant-class bias and balancing the learning signal. Robustness improves further with more descriptive language supervision, larger data scale, and broader open-world concepts. Results are validated in both supervised and self-supervised settings, enabling imbalanced-data models to reach CLIP-level performance.",{"@graph":14,"@context":73},[15,34,56],{"@type":16,"itemListElement":17},"BreadcrumbList",[18,23,27,31],{"item":19,"name":20,"@type":21,"position":22},"https://docshare.wps.com","Home","ListItem",1,{"item":24,"name":25,"@type":21,"position":26},"https://docshare.wps.com/document/","Document",2,{"item":28,"name":29,"@type":21,"position":30},"https://docshare.wps.com/document/research-report/","Research & Report",3,{"item":32,"name":10,"@type":21,"position":33},"https://docshare.wps.com/document/clip-more-robust-to-long-tailed-pre-training-data-a-controlled-study-for-transferable-insights-neurips-2024/269382/",4,{"url":32,"name":10,"@type":35,"image":36,"author":41,"headline":10,"publisher":44,"fileFormat":47,"inLanguage":8,"description":12,"dateModified":48,"datePublished":49,"encodingFormat":47,"isAccessibleForFree":50,"interactionStatistic":51},"DigitalDocument",{"url":37,"@type":38,"width":39,"height":40},"https://docshare.wps.com/thumbnails/clip-more-robust-to-long-tailed-pre-training-data-a-controlled-study-for-transferable-insights-neurips-2024/269382.png","ImageObject",300,407,{"name":42,"@type":43},"Fahsai","Person",{"url":19,"name":45,"@type":46},"DocShare","Organization","application/pdf","2026-10-17","2026-09-15",true,{"@type":52,"interactionType":53,"userInteractionCount":55},"InteractionCounter",{"@type":54},"ViewAction",13,{"@type":57,"mainEntity":58},"FAQPage",[59,65,69],{"name":60,"@type":61,"acceptedAnswer":62},"Why is CLIP more robust to long-tailed, imbalanced pre-training data than supervised learning?","Question",{"text":63,"@type":64},"Controlled analysis shows CLIP’s pretext task acts like a dynamic classification problem where only subsets of classes appear during training, which isolates bias from dominant classes and balances the learning signal.","Answer",{"name":66,"@type":61,"acceptedAnswer":67},"What experimental factors are studied to explain CLIP’s robustness?",{"text":68,"@type":64},"The paper studies factors including supervision signal and the pretext task, data distribution, scale, and open-world concepts, using controlled experiments.",{"name":70,"@type":61,"acceptedAnswer":71},"How do language descriptiveness and data scale affect CLIP’s performance?",{"text":72,"@type":64},"More descriptive language supervision increases both robustness and discriminability while preserving feature variation, and scaling up data further improves both robustness and discriminability.","https://schema.org",{"og:url":32,"og:type":75,"og:title":10,"og:site_name":45,"og:description":12},"article",{"robots":77,"canonical":32},"index,follow",{"doc_id":79,"site_id":7},269382,1789451077,{"code":4,"msg":82,"data":83},"success",[84,88,92,96,101,106,111,115,120,123,127],{"id":22,"doc_module":4,"doc_module_name":25,"category_name":85,"show_sort_weight":86,"slug":87},"Story & Novel",90,"story-novel",{"id":26,"doc_module":4,"doc_module_name":25,"category_name":89,"show_sort_weight":90,"slug":91},"Literature",80,"literature",{"id":33,"doc_module":4,"doc_module_name":25,"category_name":93,"show_sort_weight":94,"slug":95},"Exam",70,"exam",{"id":97,"doc_module":4,"doc_module_name":25,"category_name":98,"show_sort_weight":99,"slug":100},5,"Comic",60,"comic",{"id":102,"doc_module":4,"doc_module_name":25,"category_name":103,"show_sort_weight":104,"slug":105},6,"Technology",50,"technology",{"id":107,"doc_module":4,"doc_module_name":25,"category_name":108,"show_sort_weight":109,"slug":110},7,"Healthcare",40,"healthcare",{"id":112,"doc_module":4,"doc_module_name":25,"category_name":29,"show_sort_weight":113,"slug":114},8,30,"research-report",{"id":116,"doc_module":4,"doc_module_name":25,"category_name":117,"show_sort_weight":118,"slug":119},9,"Religion & Spirituality",20,"religion-spirituality",{"id":118,"doc_module":4,"doc_module_name":25,"category_name":121,"show_sort_weight":118,"slug":122},"World Cup","world-cup",{"id":124,"doc_module":4,"doc_module_name":25,"category_name":125,"show_sort_weight":124,"slug":126},10,"Lifestyle","lifestyle",{"id":128,"doc_module":4,"doc_module_name":25,"category_name":129,"show_sort_weight":97,"slug":130},19,"General","general",{"code":4,"msg":82,"data":132},{"doc_id":79,"user_id":133,"nickname":42,"user_avatar":134,"doc_module":4,"category_id":112,"category_name":29,"doc_title":10,"doc_description":12,"doc_content":135,"file_id":136,"file_url":137,"file_type":138,"file_size":139,"view_count":55,"is_deleted":4,"is_public":22,"is_downloadable":22,"audit_status":22,"page_count":140,"language":141,"language_code":8,"site_id":7,"html_lang":8,"table_of_contents":142,"faqs":143,"seo_title":144,"seo_description":12,"update_tm":80,"read_time":145,"preview_page_count":4},549768702563,"https://ap-avatar.wpscdn.com/avatar/8000c4aa63b76e948b?x-image-process=image/resize,m_fixed,w_180,h_180&k=1786536092046926083","What Makes CLIP More Robust to Long-Tailed Pre-Training Data? A Controlled Study for Transferable Insights  \nXin Wen 1 Bingchen Zhao2 Yilun Chen3 Jiangmiao Pang3† Xiaojuan Qi 1†  \n1The University of Hong Kong 2University of Edinburgh 3 Shanghai AI Laboratory  \n{wenxin, xjqi}@eee.hku.hk [pangjiangmiao@pjlab.org.cn](pangjiangmiao@pjlab.org.cn)  \nAbstract  \nSevere data imbalance naturally exists among web-scale vision-language datasets.  \nDespite this, we find CLIP pre-trained thereupon exhibits notable robustness to the data imbalance compared to supervised learning, and demonstrates significant effectiveness in learning generalizable representations. With an aim to investigate the reasons behind this finding, we conduct controlled experiments to study various underlying factors, and reveal that CLIP’s pretext task forms a dynamic classification problem wherein only a subset of classes is present in training. This isolates the bias from dominant classes and implicitly balances the learning signal. Furthermore, the robustness and discriminability of CLIP improve with more descriptive language supervision, larger data scale, and broader open-world concepts, which are inaccessible to supervised learning. Our study not only uncovers the mechanisms behind CLIP’s generalizability beyond data imbalance but also provides transferable insights for the research community. The findings are validated in both supervised and self-supervised learning, enabling models trained on imbalanced data to achieve CLIP-level performance on diverse recognition tasks. Code and data are available at: [https://github.com/CVMI-Lab/clip-beyond-tail](https://github.com/CVMI-Lab/clip-beyond-tail).  \n1 Introduction  \nThe development of contrastive language-image pre-training (CLIP) [36, 44, 57, 68, 93] has demonstrated unprecedented success in learning generalizable representations, empowering zero-shot vision tasks and robustness to natural distributional shifts. This success can be primarily attributed to the effective use of large-scale uncurated image captioning datasets collected from the web. A recent trend involves delving into the distribution of these datasets and explicitly introducing interventions to the curation process to create better data for training [29, 91] . However, limited research has been conducted on analyzing the distribution of concepts/classes in these datasets and the behavior of CLIP under varying distributions. This work thus starts by presenting a concept-centric analysis of existing web-scale image-text datasets and models pre-trained accordingly (Fig. 1) .  \nMotivation. Our motivation for this study arises from an intriguing observation of CLIP’s zero-shot performance on ImageNet: CLIP is notably more robust to pre-trained data imbalance than supervised learning. We examine various vision-language datasets at different scales, and analyze their distribution with respect to ImageNet classes. We find that image-text datasets share an extremely imbalanced class distribution (Fig. 1a) . Interestingly, we find that the zero-shot classification performance of trained CLIP models is more robust to this imbalance, especially compared to models obtained by  \nsupervised learning. This is evidenced by a weaker correlation between a class’s performance and its †Corresponding author.  \n38th Conference on Neural Information Processing Systems (NeurIPS 2024) .  \n(a) Class frequencies (log scale) ranked by LAION-400M.  \nMetaCLIP-2.5B  \nLAION-2B  \nMetaCLIP-400M  \nLAION-400M  \nYFCC-15M  \nCC-12M  \nLAIONet-3M  \n0.0 0.2 0.4 0.6 0.8 Spearman [corr. to](corr. to) per-class frequency  \n(b) Correlation between class-wise statistics.  \nFigure 1: Per-class statistics of image-text datasets and models trained on top. (a) A highly imbalanced class distribution is shared across datasets.‡(b) Compared to supervised learning (✖ SL), CLIP’s performance (measured by • accuracy) is less biased by data frequency, and the classifier is notably uncorrelated (measure","cbCaip6a8juGwpLV","https://ap.wps.com/l/cbCaip6a8juGwpLV","pdf",2159334,35,"English","# Abstract\n# Introduction\n## Motivation\n## Our study and findings\n# Applications","[{\"question\":\"Why is CLIP more robust to long-tailed, imbalanced pre-training data than supervised learning?\",\"answer\":\"Controlled analysis shows CLIP’s pretext task acts like a dynamic classification problem where only subsets of classes appear during training, which isolates bias from dominant classes and balances the learning signal.\"},{\"question\":\"What experimental factors are studied to explain CLIP’s robustness?\",\"answer\":\"The paper studies factors including supervision signal and the pretext task, data distribution, scale, and open-world concepts, using controlled experiments.\"},{\"question\":\"How do language descriptiveness and data scale affect CLIP’s performance?\",\"answer\":\"More descriptive language supervision increases both robustness and discriminability while preserving feature variation, and scaling up data further improves both robustness and discriminability.\"}]","CLIP More Robust to Long-Tailed Pre-Training Data - A Controlled Study for Transferable Insights - NeurIPS 2024 | PDF",88]